Systems and methods for configurable operation of a robot based on area classification

ABSTRACT

A method of operating a mobile robot includes generating a segmentation map defining respective regions of a surface based on occupancy data that is collected by a mobile robot responsive to navigation of the surface, identifying sub-regions of at least one of the respective regions as non-clutter and clutter areas, and computing a coverage pattern based on identification of the sub-regions. The coverage pattern indicates a sequence for navigation of the non-clutter and clutter areas, and is provided to the mobile robot. Responsive to the coverage pattern, the mobile robot sequentially navigates the non-clutter and clutter areas of the at least one of the respective regions of the surface in the sequence indicated by the coverage pattern. Related methods, computing devices, and computer program products are also discussed.

CLAIM OF PRIORITY

This application is a continuation application of and claims priorityfrom U.S. patent application Ser. No. 16/388,286, filed Apr. 18, 2019,which application is a continuation of and claims priority to U.S.patent application Ser. No. 15/943,083, filed Apr. 2, 2018, issued onJun. 4, 2019 as U.S. Pat. No. 10,310,507, which is a continuation ofU.S. patent application Ser. No. 15/704,814, filed Sep. 14, 2017, issuedon Jan. 1, 2019 as U.S. Pat. No. 10,168,709, which claims the benefit ofpriority from U.S. Provisional Patent Application No. 62/394,638,entitled “SYSTEMS AND METHODS FOR CONFIGURABLE OPERATION OF A ROBOTUSING A REMOTE USER INTERFACE” and filed Sep. 14, 2016, in the UnitedStates Patent and Trademark Office, the disclosures of each areincorporated by reference herein.

FIELD

The present invention relates generally to robotic systems and, morespecifically, to configuring the operation of autonomous robots.

BACKGROUND

Automated robots and robotic devices are used to perform taskstraditionally considered mundane, time-consuming, or dangerous. Forexample, substantially autonomous robots may be used to provide coverageof a surface to perform operations such as cleaning, performing surfacetreatments and/or painting. However, from a user's perspective,autonomous robot missions may be unpredictable leaving the user with alack of understanding regarding which area the robot will move to next.Additionally, the user may lack any information and/or input regarding amission.

SUMMARY

Some embodiments of the present invention include a computing deviceincluding a processor and a memory coupled to the processor, where thememory includes a non-transitory computer-readable storage mediumstoring computer-readable program code therein. The computer-readableprogram code is executable by the processor to perform operationsincluding generating a segmentation map defining respective regions of asurface based on occupancy data that is collected by a mobile robotresponsive to navigation of the surface, classifying or otherwiseidentifying sub-regions of at least one of the respective regions asfirst and second areas (such as non-clutter and clutter areas) based ondifferences therein, and computing a coverage pattern based onidentification of the sub-regions. The coverage pattern indicates asequence for navigation of the first and second areas (such as thenon-clutter and clutter areas), and is provided to the mobile robot.Responsive to the coverage pattern, the mobile robot sequentiallynavigates the first and second areas (for example, the non-clutter andclutter areas) of the at least one of the respective regions of thesurface in the sequence indicated by the coverage pattern.

Some embodiments of the present invention include a method of operatinga mobile robot. The method includes executing, by at least oneprocessor, computer readable instructions stored in a non-transitorycomputer readable storage medium to perform operations includinggenerating a segmentation map defining respective regions of a surfacebased on occupancy data that is collected by the mobile robot responsiveto navigation of the surface, classifying or otherwise identifyingsub-regions of at least one of the respective regions as non-clutter andclutter areas, and computing a coverage pattern based on identificationof the sub-regions. The coverage pattern indicates a sequence fornavigation of the non-clutter and clutter areas, and is provided to themobile robot. Responsive to the coverage pattern, the mobile robotsequentially navigates the non-clutter and clutter areas of the at leastone of the respective regions of the surface in the sequence indicatedby the coverage pattern.

Some embodiments of the present invention include a computer programproduct for operating a mobile robot. The computer program productincludes a non-transitory computer readable medium having computerreadable instructions stored therein, that, when executed by aprocessor, causes a processor to perform operations including generatinga segmentation map defining respective regions of a surface based onoccupancy data that is collected by the mobile robot responsive tonavigation of the surface, classifying or otherwise identifyingsub-regions of at least one of the respective regions as non-clutter andclutter areas, and computing a coverage pattern based on identificationof the sub-regions. The coverage pattern indicates a sequence fornavigation of the non-clutter and clutter areas, and is provided to themobile robot. Responsive to the coverage pattern, the mobile robotsequentially navigates the non-clutter and clutter areas of the at leastone of the respective regions of the surface in the sequence indicatedby the coverage pattern.

In some embodiments, the computing device may be a component of themobile robot, and the occupancy data may be detected by at least onesensor of the mobile robot. The operations may further include operatinga drive of the mobile robot to sequentially navigate the non-clutter andclutter areas of the at least one of the respective regions of thesurface in the sequence indicated by the coverage pattern.

In some embodiments, the computing device may be a component of a servercommunicatively coupled to the mobile robot. The operations may furtherinclude receiving the occupancy data from the mobile robot responsive tothe navigation of the surface, and storing the segmentation map in adata store.

In some embodiments, the sequence may further include navigation of aperimeter of the surface, and the mobile robot may sequentially navigatethe non-clutter area, the clutter area, and the perimeter of the surfacein the sequence indicated by the coverage pattern.

In some embodiments, the coverage pattern may further indicate a rankdirection for the non-clutter area. The rank direction may correspond toa dimension of the at least one of the respective regions that is longerthan another dimension thereof, and the mobile robot may traverse thenon-clutter area in the rank direction indicated by the coveragepattern.

In some embodiments, the respective regions defined by the segmentationmap may correspond to respective rooms. The coverage pattern may furtherindicate an order of navigation of the respective rooms, and the mobilerobot may sequentially navigate the non-clutter and clutter areas of oneof the respective rooms in the sequence indicated by the coveragepattern before navigation of a next one of the respective rooms in theorder indicated by the coverage pattern.

In some embodiments, computing the coverage pattern may further includedetermining a location on the surface for ending the navigation of thenon-clutter area based on a proximity to the clutter area. Responsive totraversing the non-clutter area in the rank direction indicated by thecoverage pattern, the mobile robot may end the navigation of thenon-clutter area at the location having the proximity to the clutterarea and may traverse the clutter area in a random pattern. Responsiveto traversing the clutter area, the mobile robot may not traverse thenon-clutter area in the rank direction before traversing at least oneother of the sub-regions.

In some embodiments, the operations may further include providing thesegmentation map to a user device. The user device may include a displayor other user interface that is configured to display the segmentationmap for viewing by a user. The operations may also include receiving,from the user device, a selection input responsive to providing thesegmentation map thereto, and computing the coverage pattern based onthe selection input.

In some embodiments, the selection input may include respective levelsof cleaning for the at least one of the respective regions and/or thesub-regions thereof. The coverage pattern may be generated to indicatethe respective levels of cleaning, and the mobile robot may execute anumber of cleaning passes in navigating the at least one of therespective regions and/or the sub-regions thereof according to therespective levels of cleaning indicated by the coverage pattern.

In some embodiments, the selection input may include a user-definedboundary or label for one or more of the respective regions and/orsub-regions thereof. The coverage pattern may be generated to indicatethe user-defined boundary or label, and the mobile robot may avoidnavigation of the one or more of the respective regions and/orsub-regions thereof according to the user-defined boundary or labelindicated by the coverage pattern.

In some embodiments, the operations may further include detectingrespective flooring types of the sub-regions, and computing the coveragepattern may further include determining the sequence for navigation ofthe non-clutter and clutter areas based on the respective flooring typesof the sub-regions corresponding thereto.

In some embodiments, the segmentation map may include data previouslyreceived from a user device and/or collected by the mobile robotresponsive to at least one previous navigation of the surface.

In some embodiments, generating the segmentation map may includecomparing the occupancy data collected by the mobile robot responsive tothe navigation of the surface with the data collected by the mobilerobot responsive to the at least one previous navigation of the surface,and updating the segmentation map to include commonalities indicated bythe comparing and to exclude outliers indicated by the comparing.

In some embodiments, generating the segmentation map may includecomputing a binary image based on the occupancy data by applying athresholding function, and performing a watershed transformation on thebinary image to partition the binary image into the respective regionsby applying a distance transform.

In some embodiments, the segmentation map may include simplifiedboundaries relative to actual boundaries indicated by the occupancy datacollected by the mobile robot responsive to navigation of the surface.

In some embodiments, the operations may further include modifying thesegmentation map having the simplified boundaries to indicate theclutter areas responsive to identification thereof prior to providingthe segmentation map to a user device.

In some embodiments, identifying the sub-regions as non-clutter andclutter areas may further include accessing a data store comprising aplurality of patterns and identifications thereof, and classifying pixelregions within the sub-regions based on similarities to the plurality ofpatterns stored in the data store.

In some embodiments, one or more of the identifications of the pluralityof patterns may be based on respective labeling inputs received from oneor more user devices.

In some embodiments, identifying the sub-regions as non-clutter andclutter areas may further include classifying the pixel regions asstatic structures corresponding to walls or dynamic structurescorresponding to clutter responsive to accessing the data store.

In some embodiments, identifying the sub-regions as non-clutter andclutter areas may further include identifying at least one boundarybased on the occupancy data, and prior to accessing the data store,distinguishing the pixel regions as attached pixel regions that areadjacent the at least one boundary and detached pixel regions that arespaced apart from the at least one boundary.

Some embodiments of the present invention include methods for operatinga mobile floor cleaning robot. Methods may include generating asegmentation map of a surface of an enclosed space using the mobilefloor cleaning robot to identify a plurality of regions of the surfacebased on an occupancy grid that is generated by the mobile floorcleaning robot, sending the segmentation map of the surface to a userdevice that includes a display that is configured to display thesegmentation map of the surface, and receiving a modified segmentationmap from the user device based on inputs received from the user into theuser device.

Some embodiments include performing a cleaning operation on the surfaceby sequentially performing the cleaning operation in individual ones ofthe plurality of regions, wherein the cleaning operation in a given oneof the plurality of regions is completed before starting the cleaningoperation in another of the plurality of regions.

In some embodiments, sending the segmentation map comprises wirelesslytransmitting segmentation map data using at least one wirelesscommunication link. Some embodiments provide that the at least onewireless communication link includes a local area network that providescommunication between the user device and the mobile floor cleaningrobot. In some embodiments, the at least one wireless communication linkcomprises a server that is configured to provide a persistent storage ofthe occupancy grid, the segmentation map and/or a modified segmentationmap.

Some embodiments provide that the modified segmentation map identifies asubset of the plurality of regions to be cleaned, the subset including aportion of one of the plurality of regions, and the modifiedsegmentation map identifies an order in which the plurality of regionsare to be cleaned.

Some embodiments include performing a cleaning operation on the surfaceby performing the cleaning operation based on the modified segmentationmap. In some embodiments, performing the cleaning operation based on themodified segmentation map comprises performing the cleaning operation ona subset of the plurality of regions. In some embodiments, performingthe cleaning operation based on the modified segmentation map comprisesperforming the cleaning operation of ones of the plurality of regionseach at a respective one of a plurality of levels that are selected bythe user and in a rank direction that corresponds to a largest dimensionof area in the respective region without obstacles, and the plurality oflevels are associated with a number of cleaning passes the mobile floorcleaning robot makes in the ones of the plurality of regions.

In some embodiments, generating the occupancy grid by exploring thesurface with the mobile floor cleaning robot comprises generating visualmaps for localization of the mobile floor cleaning robot within theoccupancy. Some embodiments provide that generating the visual maps forlocalization comprises detecting a feature using a plurality of imagescaptured by the mobile floor cleaning robot, determining a featuredescription of the feature based on the image captured by the mobilefloor cleaning robot, classifying the feature by performing a featurelookup in a feature database, determining a pose and position of themobile floor cleaning robot that is associated with the plurality ofimages and generating a landmark in the occupancy grid that isassociated with the pose and the position of the mobile floor cleaningrobot. Some embodiments provide that the method further comprisesaugmenting the occupancy grid with updated landmark data that isgenerated over multiple missions of the mobile floor cleaning robot.

In some embodiments, the occupancy grid comprises a plurality of pixelseach having a grayscale value that corresponds to whether the pixellocation is occupied, traversable or unexplored. Some embodimentsprovide that generating the segmentation map of the surface using theoccupancy grid to identify the plurality of regions of the surfacecomprises applying a median filter to the plurality of pixels thatreplaces each pixel value with a median value corresponding to pixelsthat surround the corresponding one of the plurality of pixels togenerate a filtered occupancy grid, computing a binary image of thefiltered occupancy grid by applying a thresholding function to thefiltered occupancy grid, and performing a watershed transformation onthe binary image to partition the binary image into the plurality ofregions and vertices of the plurality of regions by applying a distancetransform.

Some embodiments include performing a clutter removal operation on theplurality of regions in the binary image. In some embodiments,performing the clutter removal operation comprises identifying aboundary of the binary image, identifying and removing pixels thatcorrespond to obstacles that are detached from the boundary of thebinary image, and identifying and removing pixels that correspond toobstacles that are attached to the boundary region of the binary image.Some embodiments include merging the vertices of the plurality ofregions to join the plurality of regions and to define corridors thatare adjacent the plurality of regions.

Some embodiments include generating an automated floor coverage patternthat is based on the segmentation map that includes the plurality ofregions, the automated floor coverage pattern including a selection of arank direction corresponding to each of the plurality of regions. Insome embodiments, the rank direction defines a dimension of the regionthat is longer than other dimensions of the region. Some embodimentsprovide that the rank direction defines a dimension of a portion of therespective region that is without obstacles and that is different from adimension of the region that is longer than other dimensions of theregion. In some embodiments, the automated floor coverage patternincludes, for each region of the plurality of regions, sequentiallyselecting an uncluttered portion of the region for coverage, a perimeterof the region for coverage and a cluttered portion of the region forcoverage.

Some embodiments include sending the segmentation map of the surface toa remote computing device that is configured to communicate with themobile floor cleaning robot and with a user mobile terminal thatincludes a display that is configured to display the segmentation map ofthe surface. In some embodiments, sending the segmentation map of thesurface to the remote computing device is performed responsive to thegeneration of the segmentation map to maintain historical datacorresponding to the mobile floor cleaning robot. Some embodimentsinclude receiving a modified segmentation map from the remote computingdevice based on data sent from the mobile floor cleaning robot and/orinputs received from a user into the user device. Some embodimentsinclude performing a cleaning operation on the surface by performing thecleaning operation based on the modified segmentation map. In someembodiments, the segmentation map includes operational datacorresponding to the mobile floor cleaning robot. Some embodimentsprovide that the operational data comprises a cleaning function map thatidentifies debris collection that corresponds to a quantity of debristhat is collected by the mobile floor cleaning robot as a function oflocation on the segmentation map.

Some embodiments of the present invention include a method of operatinga user terminal, including executing, by at least one processor of theuser terminal, computer readable instructions stored in a memorycomprising a non-transitory computer readable storage medium to performoperations including receiving, via a transceiver of the user terminal,a segmentation map defining respective regions of a surface based onoccupancy data that is collected by a mobile robot responsive tonavigation of the surface. The segmentation map identifies sub-regionsof at least one of the respective regions as clutter and non-clutterareas. The operations further include displaying, via a user interfaceof the user terminal, a graphical representation of the segmentationmap, receiving, via the user interface, a selection input correspondingto one or more of the respective regions of the segmentation map and/orthe sub-regions thereof, and transmitting, via the transceiver, amodified segmentation map including the selection input to the mobilerobot.

Some embodiments of the present invention include computer programproducts that include a computer readable storage medium having computerreadable program code embodied in the medium that when executed by aprocessor of a computer causes the computer to perform operationscomprising receiving a segmentation map of a surface of an enclosedspace, the segmentation map including a plurality of regions of thesurface and generated by a mobile floor cleaning robot, displaying animage of the segmentation map on a display of the computer, receiving,via a user input device of the computer, a user input corresponding tothe segmentation map of the surface, and sending, to the mobile floorcleaning robot, a modified segmentation map that is modifiedcorresponding to the user input.

In some embodiments, receiving the user input comprises receiving theuser input to modify an appearance of the image of the segmentation map.Some embodiments provide that the user input to modify the appearance ofthe image includes a user input to change an angle of perspective of theuser relative to the image, a dimensional selection, a zoom selection,and a map feature selection. In some embodiments, the map featureselection comprises a heat map selection that causes the image of thesegmentation map displayed to illustrate a level of historical debriscollection as a function of location on the segmentation map. Someembodiments provide that the map feature selection comprises a cluttermap selection that causes the image of the segmentation map displayed toselectively illustrate clutter on the segmentation map.

In some embodiments, receiving the user input comprises receiving adivide region input that is configured to cause one of the plurality ofregions to divide into more than one region.

Some embodiments provide that receiving the user input comprisesreceiving a combine region input that is configured to cause more thanone of the plurality of regions to combine into a single region.

In some embodiments, operations further comprise receiving, by thecomputer, a status communication that is generated by the mobile floorcleaning robot and responsive to receiving the status communication,displaying a message on the display of the computer. In someembodiments, the status communication comprises a progress updatecorresponding to a cleaning mission. Some embodiments provide that theprogress update corresponding to the cleaning mission is displayed as ananimated graphic of a cleaning operation on the image of thesegmentation map. In some embodiments, the progress update is asubstantially real-time progress update that includes a visualindication on the image of the segmentation map that is displayed.

In some embodiments, the status communication comprises an alert messagethat identifies a condition that is detected by the mobile floorcleaning robot. Some embodiments provide that the condition comprises anincomplete mission condition and the mobile floor cleaning robot isunable to complete a cleaning mission.

In some embodiments, the condition comprises an unexpected obstaclecondition, the mobile floor cleaning robot detects an obstacle that isnot included in the segmentation map, and a location of the unexpectedobstacle is displayed on the image of the segmentation map.

Some embodiments provide that receiving the user input comprisesreceiving the user input that includes a cleaning mission instruction.In some embodiments, the cleaning mission instruction comprises acleaning order instruction that includes an identification of which onesof the plurality of regions are to be cleaned in a given mission and acleaning order corresponding to identified ones of the plurality ofregions. In some embodiments, the cleaning mission instruction includesa first cleaning mission instruction that identifies a first selectionof the plurality of regions to be cleaned and a second cleaning missionthat identifies a second selection of the plurality of regions to becleaned that is different from the first selection of the plurality ofregions. Some embodiments provide that the first cleaning missioninstruction includes a first scheduled time and the second cleaningmission includes a second scheduled time that is different from thefirst scheduled time.

In some embodiments, the cleaning mission instruction comprises a firstcleaning level for a first region of the plurality of regions and asecond cleaning level that is different from the first cleaning levelfor a second region of the plurality of regions, the first cleaninglevel identifies a first quantity of times that the mobile floorcleaning robot cleans the first region, and the second cleaning levelidentifies a second quantity of times that the mobile floor cleaningrobot cleans the second region.

In some embodiments, the cleaning mission instruction comprises anidentification of a portion of the occupancy floor space that is not tobe cleaned by the mobile floor cleaning robot. Some embodiments providethat the identification of the portion of the occupancy floor space thatis not to be cleaned by the mobile floor cleaning robot is persistent.In some embodiments, the identification of the portion of the occupancyfloor space that is not to be cleaned by the mobile floor cleaning robotapplies to a single cleaning mission.

Some embodiments provide that the cleaning mission instruction comprisesa primary cleaning direction for each of the plurality of regions thatidentifies a cleaning direction that the mobile floor cleaning robotcleans in obstacle free portions of a cleaning operation.

In some embodiments, receiving the user input comprises receiving a userpreference data input that includes a request for user preference dataand operations further comprise displaying user preference data. Someembodiments provide that displaying user preference data comprisesdisplaying historical user preference data and/or user preference datathat corresponds to a single cleaning mission.

Some embodiments provide that receiving the user input comprisesreceiving a robot team configuration input that includes a firstcleaning mission instruction corresponding to a first mobile floorcleaning robot and a second cleaning mission corresponding to a secondmobile floor cleaning robot. In some embodiments, the first and secondcleaning missions correspond to the segmentation map of the surface ofthe occupancy. Some embodiments provide that the first and secondcleaning missions correspond to different segmentation maps of differentportions of the occupancy. In some embodiments, the first mobile floorcleaning robot performs a first type of cleaning operation and thesecond mobile floor cleaning robot performs a second type of cleaningoperation.

Some embodiments provide that the computer is a hand-held mobileterminal and the display is a touch-screen display. Some embodimentsprovide that the operations further comprise generating a graphical userinterface via the touch-screen display and receiving the user input viathe user input device comprises receiving the user input via thegraphical user interface using the touch-screen.

In some embodiments, sending the modified segmentation map to the mobilefloor cleaning robot comprises sending the modified segmentation map toa remote server that is configured to provide a persistent storage themodified segmentation map.

Some embodiments of the present invention include a robot managementnode comprising a processor and a memory coupled to the processor andcomprising computer readable program code that when executed by theprocessor causes the processor to perform operations. The operationsinclude receiving a segmentation map of a surface of an enclosed spacefrom a robot, the segmentation map identifying a plurality of regions ofthe surface based occupancy data that is collected by the robot andstoring the segmentation map of the surface of the occupancy in a datastore that is associated with the robot.

In some embodiments, operations further comprise receiving user inputdata from a user device that receives a user input corresponding to thesegmentation map. In some embodiments, the segmentation map is modifiedbased on the user input data to generate a modified segmentation map. Insome embodiments, operations further include sending the modifiedsegmentation map based on the user input data and the segmentation mapto the robot that is operable to perform a cleaning mission responsiveto the modified segmentation map.

In some embodiments, receiving the segmentation map from the robotcomprises receiving multiple segmentation maps that are generated by therobot during multiple cleaning missions, the segmentation map is updatedand/or augmented by later received segmentation maps, and storing thesegmentation map comprises storing the multiple segmentation maps togenerate historical segmentation map data including the modifiedsegmentation map.

Some embodiments provide that the modified segmentation map identifies asubset of the plurality of regions to be cleaned, the subset including aportion of one of the plurality of regions, and the modifiedsegmentation map identifies an order in which the plurality of regionsare to be cleaned.

Some embodiments provide that the cleaning mission includes performing acleaning operation of ones of the plurality of regions each at arespective one of a plurality of levels that are selected by the userand in a rank direction that corresponds to a largest dimension of areain the respective region without obstacles and the plurality of levelsare associated with a number of cleaning passes the robot makes in theones of the plurality of regions.

In some embodiments, the rank direction defines a dimension of theregion that is longer than other dimensions of the region.

Some embodiments provide that the rank direction defines a dimension ofa portion of the respective region that is without obstacles and that isdifferent from a dimension of the region that is longer than otherdimensions of the region.

In some embodiments, the segmentation map includes operational datacorresponding to the robot. Some embodiments provide that theoperational data comprises a cleaning function map that identifiesdebris collection that corresponds to a quantity of debris that iscollected by the robot as a function of location on the segmentationmap. In some embodiments, the operational data comprises a cleaningfunction map that identifies a quantity of passes that are performed bythe robot as a function of location on the segmentation map.

In some embodiments, receiving user input data comprises receiving arobot team configuration input that includes a first cleaning missioninstruction corresponding to a robot and a second cleaning missioncorresponding to a second robot.

Some embodiments provide that the processer and the memory are includedin a cloud-based server that is remote from the robot and the occupancy.

Some embodiments are directed to an electronic device that is operableto performed operations described herein.

It is noted that aspects of the invention described with respect to oneembodiment, may be incorporated in a different embodiment although notspecifically described relative thereto. That is, all embodiments and/orfeatures of any embodiment can be combined in any way and/orcombination. These and other objects and/or aspects of the presentinvention are explained in detail in the specification set forth below.

Further features, advantages and details of the present invention willbe appreciated by those of ordinary skill in the art from a reading ofthe figures and the detailed description of the embodiments that follow,such description being merely illustrative of the present invention.Other methods, devices, and computer program products according toembodiments will be or become apparent to one with skill in the art uponreview of the following drawings and detailed description. It isintended that all such additional methods, devices, and computer programproducts be included within this description, be within the scope of thepresent inventive subject matter, and be protected by the accompanyingclaims. Moreover, it is intended that all embodiments disclosed hereincan be implemented separately or combined in any way and/or combination.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects of the present invention are illustrated by way of example andare not limited by the accompanying drawings. In the drawings

FIG. 1 is a block diagram illustrating a system of devices that may beused to provide robot management according to some embodiments of thepresent invention.

FIG. 2A is a schematic diagram representing a mobile robot according tosome embodiments of the present invention.

FIGS. 2B and 2C are top and bottom perspective views, respectively, of amobile robot according to some embodiments of the present invention.

FIG. 3 is a block diagram of an electronic device that may be an exampleof the robot management node according to some embodiments of thepresent invention.

FIG. 4 is a block diagram of a wireless communication device and methodsof operation according to some embodiments of the present invention

FIG. 5 is a block diagram illustrating a method for operating a mobilefloor cleaning robot according to some embodiments of the presentinvention.

FIG. 6 is a block diagram illustrating operations of computer readableprogram product on a user device according to some embodiments of thepresent invention.

FIG. 7 is a block diagram illustrating operations of computer programproduct on a robot management node according to some embodiments of thepresent invention.

FIGS. 8A through 8C are graphical representations that may be displayedvia a user interface of a communication device illustrating a raw datamap, a cleaned map and a segmentation map in an automated map cleaningand segmentation operation according to some embodiments of the presentinvention.

FIGS. 9A through 9C are graphical representations that may be displayedvia a user interface of a communication device illustrating user displaypreference options of a raw data map, a segmentation map with clutterpixels and segmentation map with clutter regions in operations accordingto some embodiments of the present invention.

FIGS. 10A through 10F are graphical representations that may bedisplayed via a user interface of a communication device illustratingoperations for performing user region merging on a segmentation mapaccording to some embodiments of the present invention.

FIGS. 11A through 11C are graphical representations that may bedisplayed via a user interface of a communication device illustratingoperations for performing user region splitting on a segmentation mapaccording to some embodiments of the present invention.

FIGS. 12A through 12D are graphical representations that may bedisplayed via a user interface of a communication device illustratingoperations for performing user region labeling on a segmentation mapaccording to some embodiments of the present invention.

FIG. 13 is a graphical representation that may be displayed via a userinterface of a communication device illustrating operations fordisplaying progress of a robot on a segmentation map according to someembodiments of the present invention.

FIG. 14 is a graphical representation that may be displayed via a userinterface of a communication device illustrating operations forproviding an alert to a user of a failure condition on a segmentationmap according to some embodiments of the present invention.

FIG. 15 is a graphical representation that may be displayed via a userinterface of a communication device illustrating operations forreceiving user input to provide a cleaning sequence of a robot on asegmentation map according to some embodiments of the present invention.

FIGS. 16A through 16G are graphical representations that may bedisplayed via a user interface of a communication device illustratingoperations for map aggregation using multiple runs in the same occupancyaccording to some embodiments of the present invention.

FIGS. 17A and 17B are graphical representations that may be displayedvia a user interface of a communication device illustrating asegmentation map from the aggregating operations based on the raw datamaps illustrated in FIGS. 16A-16G that are without clutter and thatinclude clutter, respectively, according to some embodiments of thepresent invention.

FIGS. 18A and 18B are graphical representations that may be displayedvia a user interface of a communication device illustrating raw data ofan occupancy map and a three-dimensional representation of the occupancymap, respectively, based on the raw data maps illustrated in FIGS.16A-16G according to some embodiments of the present invention.

FIG. 19 is a graphical representation that may be displayed via a userinterface of a communication device illustrating an occupancy mapillustrating operations for generating persistent informationcorresponding to an enclosed space according to some embodiments of thepresent invention.

FIGS. 20A through 20K are graphical representations that may bedisplayed via a user interface of a communication device illustratingviews of occupancy maps and segmentation maps corresponding to datagathered during robot exploration according to some embodiments of thepresent invention.

FIG. 21 is a diagram illustrating a system of devices that may be usedto provide robot management according to some embodiments of the presentinvention.

FIG. 22 is a graphical representation that may be displayed via a userinterface of a communication device illustrating a history screenincluding a mission coverage map corresponding to a cleaning missionaccording to some embodiments of the present invention.

FIG. 23 is a graphical representation that may be displayed via a userinterface of a communication device illustrating an occupancy mapillustrating real-time reporting corresponding to a cleaning missionaccording to some embodiments of the present invention.

FIG. 24 is a graphical representation that may be displayed via a userinterface of a communication device illustrating an occupancy mapillustrating a multi-mission coverage display according to someembodiments of the present invention.

FIG. 25 is a graphical representation that may be displayed via a userinterface of a communication device illustrating room-based data thatmay be displayed according to some embodiments of the present invention.

FIG. 26 is a graphical representation demonstrating operations in whichuser input may be provided via a communication device using a voicecommand according to some embodiments of the present invention.

FIG. 27 is a graphical representation of receiving user input to definea cleaning mission boundary responsive to a gesture on a touch screen ofa communication device according to some embodiments of the presentinvention.

FIGS. 28A through 28C are graphical representations that may bedisplayed via a user interface of a communication device illustrating araw data map, a cleaned map and a cleaned map with detected clutterregions according to some embodiments of the present invention.

FIGS. 29A through 29D are graphical representations that may bedisplayed via a user interface of a communication device illustrating asegmentation map that includes respective area-specific cleaningoperations according to some embodiments of the present invention.

FIGS. 30A through 30G are graphical representations that may bedisplayed via a user interface of a communication device for userselection of respective area-specific cleaning operations and/or asequence of performing the area cleaning operations according to someembodiments of the present invention.

FIGS. 31A through 31C are graphical representations that may bedisplayed via a user interface of a communication device for userselection of cleaning patterns for open area cleaning operationsaccording to some embodiments of the present invention.

FIG. 32 is a graphical representation that may be displayed via a userinterface of a communication device illustrating detection and/or userselection of floor types for area-specific cleaning operations accordingto some embodiments of the present invention.

FIGS. 33A through 33D illustrate operations for generating a simplifiedmap for display via a user interface of a communication device accordingto some embodiments of the present invention.

FIGS. 34A through 34F are graphical representations that may bedisplayed via a user interface of a communication device illustratingobject detection operations according to some embodiments of the presentinvention.

FIGS. 35A through 35D are graphical representations illustratingoperations for clutter classification and feature extraction accordingto some embodiments of the present invention.

FIGS. 36A through 36C are graphical representations illustratingoperations for updating persistent map data by feature matching andsegmentation transfer between multiple maps according to someembodiments of the present invention.

FIG. 37 is a graphical representation that may be displayed via a userinterface of a communication device illustrating operation of a mobilerobot responsive to user selection of temporal status according to someembodiments of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS

The present invention now will be described more fully hereinafter withreference to the accompanying drawings, in which illustrativeembodiments of the invention are shown. In the drawings, the relativesizes of regions or features may be exaggerated for clarity. Thisinvention may, however, be embodied in many different forms and shouldnot be construed as limited to the embodiments set forth herein; rather,these embodiments are provided so that this disclosure will be thoroughand complete, and will fully convey the scope of the invention to thoseskilled in the art.

It will be understood that when an element is referred to as being“coupled” or “connected” to another element, it can be directly coupledor connected to the other element or intervening elements may also bepresent. In contrast, when an element is referred to as being “directlycoupled” or “directly connected” to another element, there are nointervening elements present. Like numbers refer to like elementsthroughout.

In addition, spatially relative terms, such as “under,” “below,”“lower,” “over,” “above,” “upper,” and the like, may be used herein forease of description to describe one element or feature's relationship toanother element(s) or feature(s) as illustrated in the figures. It willbe understood that the spatially relative terms are intended toencompass different orientations of the device in use or operation inaddition to the orientation depicted in the figures. For example, if thedevice in the figures is turned over, elements described as “under” or“beneath” other elements or features would then be oriented “over” theother elements or features. Thus, the example term “under” can encompassboth an orientation of over and under. The device may be otherwiseoriented (rotated 90 degrees or at other orientations) and the spatiallyrelative descriptors used herein interpreted accordingly.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”. “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof. As used herein the expression“and/or” includes any and all combinations of one or more of theassociated listed items.

A “mobile robot” may refer to any device including a processor, memory,and drive system for navigating variable environment conditions andmaking autonomous decisions based on a plurality of sensor inputs.Mobile robots as described herein, may include robot cleaners (such asiRobot® ROOMBA®, BRAAVA®, and/or BRAAVA Jet™ cleaners). A “communicationsignal” may refer to any signal transmitted by a network-enabledelectronic device. Such electronic devices may include a processor,memory, and a transmitter and/or receiver for communication via awireless personal, local, and/or wide area network, and may include, butare not limited to, Internet-of-Things (IoT) devices. Wirelesscommunication signals may include radio frequency signals, including butnot limited to Wi-Fi signals, Bluetooth signals, and/or optical signals.

As used herein, a “wireless communication device” includes, but is notlimited to, a device that is configured to receive/transmitcommunication signals via a wireless interface with, for example, acellular network, a wireless local area network (WLAN), a digitaltelevision network such as a DVB-H network, a satellite network, anAM/FM broadcast transmitter, and/or another communication terminal. Awireless communication device may be referred to as a “wirelesscommunication terminal,” a “wireless terminal” and/or a “mobileterminal.” Examples of wireless communication devices include, but arenot limited to, a satellite or cellular radiotelephone; a PersonalCommunications System (PCS) terminal that may combine a cellularradiotelephone with data processing, facsimile and data communicationscapabilities; a PDA that can include a radiotelephone, pager,Internet/intranet access, Web browser, organizer, calendar and/or aglobal positioning system (GPS) receiver; and a conventional laptopand/or palmtop receiver or other appliance that includes a radiotransceiver, including WLAN routers and the like.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this invention belongs. It will befurther understood that terms, such as those defined in commonly useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art andwill not be interpreted in an idealized or overly formal sense unlessexpressly so defined herein.

FIG. 1 is a block diagram illustrating a system of devices that may beused to provide robot management according to some embodiments of thepresent invention. Some embodiments provide that systems and/oroperations disclosed herein may include a robot management node 200 thatmay be communicatively coupled to one or more robots 100 and/or one ormore user devices 202 via one or more wired and/or wirelesscommunication and/or data networks 204. In some embodiments, the robot100 and the user device 202 may be directly communicatively coupled viaa near field communication protocol. The user device 202 may be a wiredor wireless communication terminal, such as a desktop or laptopcomputer, tablet, or smartphone. The user device 202 may include anapplication stored in a memory thereof for communication with therobot(s) 100 and/or the robot management node 200.

In some embodiments, the robot management node 200 may include adedicated computer that includes a running instance of an applicationthat is capable of receiving and transmitting communications to the oneor more robots 100 and/or the one or more user devices 202. Someembodiments provide that the robot management node 200 may furtherinclude instances and/or operations corresponding to a file server, adatabase server, an application server and/or a web server, includingcloud-based servers. In some embodiments, the robot management node 200includes a stationary or fixed location processing device that receivespower via a wired power connection to a facility and/or building powersource. Some embodiments provide that the robot management node 200transmits and/or receives data via a wired data connection. In someembodiments, the robot management node 200 may be provided on a mobileprocessing device that includes an on-board power source and thattransmits and/or receives data using a wireless and/or wired dataconnection. As disclosed herein, the robot management node 200 mayprovide a persistent store of segmentation maps, user inputs, robotoperational data, and the like. For example, the persistent store mayinclude maps generated from occupancy data and/or wireless signalstrength/coverage data collected by the robot(s) 100 during navigationof one or more operating environments, as described for example in U.S.patent application Ser. No. 15/588,117 entitled “METHODS, SYSTEMS, ANDDEVICES FOR MAPPING WIRELESS COMMUNICATION SIGNALS FOR MOBILE ROBOTGUIDANCE,” the disclosure of which is incorporated by reference.

FIGS. 2A-2C illustrate an autonomous coverage robot system that includesa mobile robot 100 (illustrated as a vacuum cleaning robot). The mobilerobot 100 is adapted to mate with a base station or charging dock and/oran evacuation dock. The system also includes a charging or energymanagement system and an auto-docking control system each includingcooperatively operating components of the mobile robot 100 and the dock.

A communications/guidance signal detector 152 is mounted on the topfront of the housing 106 of the mobile robot 100. The detector 152 isoperable to receive signals projected from an emitter (e.g., theavoidance signal emitter and/or homing and alignment emitters of thedock) and (optionally) an emitter of a navigation or virtual wallbeacon.

A navigational control system may be used advantageously in combinationwith the mobile robot 100 to enhance the cleaning efficiency thereof, byadding a deterministic component (in the form of a control signal thatcontrols the movement of the mobile robot 100) to the motion algorithms,including random motion, autonomously implemented by the mobile robot100. The navigational control system operates under the direction of anavigation control algorithm. The navigation control algorithm includesa definition of a predetermined triggering event. Broadly described, thenavigational control system, under the direction of the navigationcontrol algorithm, monitors the movement activity of the mobile robot100. The predetermined triggering event is a specific occurrence orcondition in the movement activity of the robot 100. Upon therealization of the predetermined triggering event, the navigationalcontrol system operates to generate and communicate a control signal tothe robot 100. In response to the control signal, the mobile robot 100operates to implement or execute a conduct prescribed by the controlsignal. i.e., the prescribed conduct. This prescribed conduct representsa deterministic component of the movement activity of the mobile robot100.

While the mobile robot 100 is vacuuming, it will periodically approachthe stationary dock. Contact with the dock could damage or move the dockinto an area that would make docking impossible. Therefore, avoidancefunctionality is desirable. To avoid inadvertent contact, the dock maygenerate an avoidance signal that may be transmitted from the emitter onthe top of the dock. The radial range of the avoidance signal from thedock may vary, depending on predefined factory settings, user settings,or other considerations. At a minimum, the avoidance signal need onlyproject a distance sufficient to protect the dock from unintentionalcontact with the mobile robot 100. The avoidance signal range can extendfrom beyond the periphery of the dock, to up to and beyond several feetfrom the dock, depending on the application.

The mobile robot 100 may be any suitable robot and associated computingdevice(s), and it will be appreciated that not all of the components,features and functionality described herein are required in mobilerobots according to embodiments of the present disclosure. Withreference to FIGS. 2A-2C, the example mobile robot 100 may include achassis 210, a controller 220, memory 222, a battery 224, a batterycharger 226, a human-machine interface (HMI) 228, a drive system 230, amapping/navigation system 240, a service operation system 242 (alsoreferred to herein as “cleaning system” and “cleaning head”), a wirelesscommunication system 250, an IR emitter 260, and environmental sensors270A-H, a debris bin 242A (to store debris collected by a cleaningoperation), a bin level sensor 242B, a dirt extraction sensor 242C (todetect the density of characteristics of the debris collected by thecleaning operation), an indicator light 274A, an audio transducer 274B,and a cleaning mode selection switch or button 274C.

The environmental sensors 270A-270H may include a camera 270B mounted ona top surface of the mobile robot 10, as shown in the top perspectiveview of FIG. 2A. The camera 270B can be used to navigate the robot 100and acquire images for other operational use. In some embodiments, thecamera 270B is a visual simultaneous location and mapping (VSLAM) cameraand is used to detect features and landmarks in the operatingenvironment and build an occupancy map based thereon.

As shown in the bottom perspective view of FIG. 2B, the mobile robot 100may further include a bumper 104, cliff sensors 195A-195D, an edge brush111 mounted or otherwise positioned at a periphery of the mobile robothousing 106. The housing 106 is illustrated in FIGS. 2A-2B as having asquared front section on which the bumper 104 is mounted; however, thehousing may have a rounded or circular shape in other embodiments.

A caster wheel 196 may be provided on the underside of the mobile robot100. In some embodiments, the caster wheel 196 may be positioned at anopposite end of the mobile robot 100 than the cleaning head 242, withthe drive rollers/tracks 232A, 232B therebetween, such that the cleaninghead 242 is a cantilevered arrangement. The mobile robot 100 may alsoinclude downward- or floor-facing camera 197.

The cameras 270B and 197 and/or other imaging sensors may collectivelyoperate as an image sensing device that can be used to acquireinformation for guidance and operation of the robot during variousoperations of the mobile robot 100. In some embodiments, the imagesensing device is used to detect obstacles and hazards about the mobilerobot 100 so that those obstacles and hazards can be avoided orotherwise addressed. Within the operational range of the image sensordevice, a downwardly directed beam can be used to detect obstacles at ornear the floor level as well as cliffs or depressions in the floor. Anupwardly directed beam can be used to detect obstacles at or above thetop of the robot 100 in order to detect and avoid obstacles under whichthe robot may become wedged. In some embodiments, the image sensingdevice is operative to effectively detect objects and voids up to atleast 10 inches forward of the mobile robot 100 and, in someembodiments, up to at least 12 inches.

The controller 220 may include any suitably configured processor orprocessors. The processor(s) may include one or more data processingcircuits, such as a general purpose and/or special purpose processor(such as a microprocessor and/or digital signal processor) that may becollocated or distributed across one or more networks. The processor isconfigured to execute program code stored in the memory 222, describedbelow as a computer readable storage medium, to perform some or all ofthe operations and methods that are described above for one or more ofthe embodiments. The memory 222 is representative of the one or morememory devices containing the software and data used for facilitatingoperations of the robot in accordance with some embodiments of thepresent disclosure. The memory 222 may include, but is not limited to,the following types of devices: cache, ROM, PROM, EPROM, EEPROM, flash,SRAM, and DRAM. The processor is thus in communication with thecontroller 200, memory 222, the cleaning system 242 and drive system230.

The drive system 230 may include any suitable mechanism or system foractively and controllably transiting the robot 100 through the livingspace 20. According to some embodiments, the drive system 230 includes aroller, rollers, track or tracks 232A, 232B and one or more onboard(i.e., carried by the mobile robot 100) electric motors 234(collectively referred to herein as a “drive” or “drive system”)operable by the controller 220 to convey the robot 100 across the floorof the operating environment 10.

The service operation system 242 may be optional in some embodiments,and is operable to execute a service operation in the living space 20.According to some embodiments, the service operation system 242 includesa floor cleaning system that cleans a floor surface of the living space20 as the robot 100 transits through the space 20. In some embodiments,the service operation system 242 includes a suction head and an onboardvacuum generator to vacuum clean the floor. In some embodiments, theservice operation system 242 includes an end effector such as (but notlimited to) a sweeping or mopping mechanism, one or more rotatingbrushes, rollers, wet or dry stationary or oscillating and/or vibratingcloths, or multilayer pad assemblies.

The wireless communication system 250 includes a wireless communicationtransceiver or module 252 and an associated antenna 254 that areconfigured to send/receive data to/from one or more computing devices toenable wireless communication between the robot 100 and the variousother connected devices in the operating environment 10, as well asnetwork segments serviced by WAPs, gateways and hubs which make up theprivate network 160, of which the mobile robot 100 constitutes a node.For example, the wireless communication transceiver or module 252 may bea Wi-Fi module. In some embodiments, the robot 100 may communicatewirelessly directly with the dock using narrowband or broadband RFcommunication. For example, if the robot 100 is not equipped with atransmitter compatible with a wireless access point (WAP), the robot 100may communicate with the dock, which may in turn relay data from therobot 100 onto the private network and onward to the intended networkentity (such as the robot management server 200).

In some embodiments, the mobile robot 100 may be generally configured inthe manner of or include features from floor cleaning robots and/orrobots as described in U.S. Pat. No. 7,024,278 and U.S. PublishedApplication No. 2007/0250212, the disclosures of which are incorporatedherein by reference, with suitable modifications. Other suitableconfigurations for the vacuum cleaning system are disclosed in U.S. Pat.No. 9,215,957 to Cohen et al., U.S. Patent Publication No. 2016/0166126to Morin et al., U.S. Pat. No. 8,881,339 to Gilbert, Jr. et al.. U.S.Patent Publication No. 2014/0222279 to Stout et al., and U.S. patentapplication Ser. No. 15/647,677 to O'Brien et al. filed Jul. 12, 2017,the disclosures of which are incorporated herein by reference. Suitablestructured light image sensing devices for use as the image sensingdevices may include the Global Shutter Image Sensor available fromPixArt Imaging, Inc. of Taiwan. The robot 100 may further include a bindetection system for sensing an amount of debris present in a cleaningbin (e.g. as described in U.S. Patent Publication No. 2012/0291809, theentirety of which is hereby incorporated by reference).

Execution of a room segmentation aims to reach optimality by avoidingunnecessary changes in direction when traversing an open environment.Also, based on predefined boundaries, frequent traveling across roomswhile cleaning can be reduced or eliminated. This makes the behaviormore understandable to the user, who may expect or desire the robot tosystematically finish one room after another.

FIG. 3 is a block diagram of an electronic computing device that may bean example of the robot management node 200 according to someembodiments of the present invention. The electronic device 300 includesone or more network interfaces 330, processor circuitry (“processor”)310, and memory 320 containing program code 322. The processor 310 mayinclude one or more data processing circuits, such as a general purposeand/or special purpose processor (e.g., microprocessor and/or digitalsignal processor) that may be collocated or distributed across one ormore networks. The processor 310 is configured to execute program code322 in the memory 320, described below as a computer readable storagemedium, to perform some or all of the operations and methods that aredescribed above for one or more of the embodiments. The memory 320 isrepresentative of the one or more memory devices containing the softwareand data used for facilitating operations for managing the robot inaccordance with some embodiments of the present invention. The memory320 may include, but is not limited to, the following types of devices:cache, ROM, PROM, EPROM, EEPROM, flash, SRAM, and DRAM.

The electronic device 300 may also include a display device 340 (whichmay display a drag-and-drop user interface) and/or an input device 350,such as a keyboard, touch sensitive display device, mouse, pointer, etc.The network interface 330 can be configured to communicate through oneor more networks with any associated available resource server(s) and/ordata repositories.

Although FIG. 3 illustrates hardware/software architectures that may beused in managing a robot according to some embodiments described herein,it will be understood that the present invention is not limited to sucha configuration but is intended to encompass any configuration capableof carrying out operations described herein.

FIG. 4 is a block diagram of a wireless communication device and methodsof operation according to some embodiments of the present invention. Thewireless communication device 400 is configured to transmit and/orreceive wireless signals over multiple wireless communicationinterfaces, and may be an example of the user device 202 describedabove. For example, a wireless communication device 400 according tosome embodiments can include a cellular communication module, aBluetooth module, an infrared communication module, a global positioningsystem (GPS) module, a WLAN module, and/or other types of communicationmodules.

In particular, the wireless communication device 400 includes a display408, such as a liquid crystal display (LCD) and/or an organic lightemitting diode (OLED) display. The wireless communication device 400 mayoptionally include a keypad 102 or other user input mechanism as a userinterface 410 of the device 400. In some embodiments, the display 408may be provided with touch screen capability to replace and/orsupplement the keypad 402.

The wireless communication device 400 may include a microphone 406 andan earphone/speaker 404. The front housing may be designed to form anacoustic seal to the user's ear when the earphone/speaker 404 is placedagainst the user's head.

The keypad 402, display 408, microphone 406, speaker 404 and camera 424may be coupled to a processor 427, such as a microprocessor ormicrocontroller, which may be configured to control operations of thewireless communication device 400. The wireless communication device 400may further include a transceiver 440 and a memory 428 coupled to theprocessor 427. Other electronic circuitry, such as a WLAN communicationinterface, a Bluetooth interface, a GPS interface, a digital signalprocessor, etc., may also be included in the electronic circuitry of thedevice 400.

The memory 428 may be a general purpose memory that is used to storeboth program instructions for the processor 427 as well as data, such asaudio data, video data, configuration data, and/or other data that maybe accessed and/or used by the processor 427. The memory 428 may includea nonvolatile read/write memory, a read-only memory and/or a volatileread/write memory. In particular, the memory 428 may include a read-onlymemory in which basic operating system instructions are stored, anon-volatile read/write memory in which re-usable data, such asconfiguration information, directory information, and other informationmay be stored, as well as a volatile read/write memory, in whichshort-term instructions and/or temporary data may be stored.

The transceiver 440 typically includes a transmitter circuit 442, areceiver circuit 444, and a modem 446, which cooperate to transmit andreceive radio frequency signals to remote transceivers via an antennaarray 450. The radio frequency signals transmitted between the device400 and the remote transceivers may comprise both traffic and controlsignals (e.g., paging signals/messages for incoming calls), which areused to establish and maintain communication with another party ordestination. More particularly, the transceiver 440, in operationalcooperation with the processor 427, may be configured for communicationaccording to multiple radio access technologies. The radio accesstechnologies may include, but are not limited to, WLAN (e.g., 802.11),WiMAX (Worldwide Interoperability for Microwave Access), 3GPP LTE (3rdGeneration Partnership Project Long Term Evolution), Universal MobileTelecommunications System (UMTS), Global Standard for Mobile (GSM)communication, General Packet Radio Service (GPRS), enhanced data ratesfor GSM evolution (EDGE), DCS, PDC, PCS, code division multiple access(CDMA), wideband-CDMA, and/or CDMA2000. Other radio access technologiesand/or frequency bands can also be used in embodiments according to theinvention.

FIG. 5 is a block diagram illustrating a method for operating a mobilefloor cleaning robot according to some embodiments of the presentinvention. The method includes generating a segmentation map of asurface of an enclosed space or other operating environment using themobile floor cleaning robot 100 to identify multiple regions of thesurface (block 504). In some embodiments, an enclosed space may includean occupancy, whether occupied or not, including, for example, aresidential building, a commercial building, a storage building, amanufacturing building and/or a portion thereof.

In some embodiments, the segmentation map is generated from an occupancygrid that is generated by the mobile floor cleaning robot during one ormore exploration and/or cleaning missions, more generally referred toherein as navigation. For example, the occupancy grid may be generatedby exploring the surface of the enclosed space with the mobile floorcleaning robot by generating one or more visual maps for localization ofthe mobile floor cleaning robot within the enclosed space. In someembodiments, the visual maps may be generated by detecting featuresusing multiple images that may be captured by one or more cameras in themobile floor cleaning robot. For example, visual features (VSLAM)generated from the video images may be used in order build the “visualmap” used to localize the robot. For each feature, a feature descriptionmay be determined and the feature may be classified by performing afeature lookup in a feature database. The occupancy grid may beintegrated with the visual map, and may further include informationgenerated from bumper hits or collisions (“bumping” events) and/orIR/PixArt data (i.e., time of flight and/or ranging sensor data forobstacle detection).

Some embodiments provide that extracted regions are then merged intocorridors and rooms according to heuristic decision rules. Then for eachregion, starting locations and directions are calculated for systematicroom cleaning. Finally, regions are systematically cleaned by the robotaccording to the shortest sequence computed by route optimization and/orby a sequence selected by a user. Final segmentation is then used on therobot for systematic room-by-room cleaning. In some embodiments, bothautomatic and user-guided segmentation results are stored in the Cloudin order to incrementally evaluate and improve the segmentationprocedure. In other embodiments, the automatic and/or user-guidedsegmentation results may be stored in a memory of the robot or at alocal hub, for example, for sharing with one or more other robots thatare local to the operating environment.

A pose and/or position of the mobile floor cleaning robot may bedetermined and associated with the corresponding images. A landmark inthe occupancy grid may be generated based on the images. The landmarkmay be associated with the pose and/or the position of the mobile floorcleaning robot. Some embodiments provide that the occupancy grid may beaugmented and/or updated with updated landmark data that is generatedover multiple missions of the mobile floor cleaning robot. Features mayalso be generated from the grid map, for example, to transfer anexisting occupancy grid to a recently generated one. This may be used totransfer existing room segmentation to a newly generated map. That is,embodiments of the present invention may utilize multiple mechanisms incombination, such as, but not limited to, robot mapping and localizationbased on visual features, and transfer of segmentation based on featuresgenerated from the occupancy grid.

In some embodiments, the occupancy grid includes multiple pixels eachhaving a grayscale value that corresponds to whether the pixel locationcorresponds to a surface location that is occupied, traversable orunexplored. As used herein, the term “traversable” may refer to anysurface location in the enclosed space that is traversable by a robot asdisclosed herein. Some embodiments provide that the segmentation map maybe generated by applying a median filter to the pixels. In someembodiments, the median filter may replace each pixel value with amedian value corresponding to pixels that surround the correspondingpixel. For example, some non-limiting embodiments provide that themedian value is determined based on the values of the pixels that arewithin five pixels of each pixel. In this manner, a filtered occupancygrid may be generated.

A binary image of the filtered occupancy grid may be computed byapplying a thresholding function to the filtered occupancy grid. Awatershed transformation may be performed on the binary image topartition the binary image into multiple regions and to identify thevertices of each of the regions by applying a distance transform.

Some embodiments further include performing a clutter removal operationon the regions in the binary image. For example, removing clutter may beperformed by identifying a boundary of the binary image, identifying andremoving pixels that correspond to obstacles that are detached from theboundary of the binary image and identifying and removing pixels thatcorrespond to obstacles that are attached to the boundary region of thebinary image, for example, by accessing a data store including groundtruth information generated at least in part from labeling inputsreceived from multiple users as described in greater detail below withreference to FIGS. 35A-35D. Some embodiments further include merging thevertices of the regions to join the regions and to define corridors thatare adjacent the regions.

Some embodiments provide further processing operations that identifyclutter in a map represented by the image M First, a boundary of animage X is identified by taking the distance transform to its inverse.The boundary may refer to the connected set of all obstacle pixelsstarting at pixel (1. 1). Note that for a map M it can be assumed thatall pixels on the edges of the image have value 1 after thresholding,i.e., the map is bounded by either obstacles or unexplored regions.Detached obstacles R_(det), i.e., obstacles that are not connected tothe boundary, are then determined. The attached obstacles R_(alt) arecomputed by segmenting the boundary using watershed segmentation ws(.)and removing regions that contains pixels on the edges of the originalimage. Algorithm 1, which follows, describes the computation of theobstacle regions from a binary image in detail:

Algorithm obstExt(X) | DT ← dt (X) | C ← ∅, Q ← ∅ // Closed list andqueue | Q.PUSH((1,1)) // Initial pixel | // Initialize Boundary | for u← 1 . . . n do | | for v ← 1 . . . m do | | | B(

) ← 0 | while Q ≠ ∅ do | | p ← Q.POP() | | C ← C ∪ {p} | | B(p) ← 1 | |if DT(p) ≥ 1 then | | | for q ∈ Neighbors(p) do | | | | if q ∉ C then || | | | Q.PUSH(q) |

_(det) ← cc_(R)(X − B) |

_(att) ← cc_(R)(ws(B)) |

_(bou) ← {r | r ∈

 

 (x,y) ∈ r,x = 1 or y = 1} |

 

 ←

_(att) \

_(bou) |

_(obs) ←

_(att) ∪

_(det)

indicates data missing or illegible when filed

The region set R_(obs) contains the obstacles which can be discriminatedbetween clutter and walls. The following filter rules distinguishbetween walls and clutter, i.e., R_(obs) R₃ is the clutter and R₃contains no clutter.

R ₁ =R _(obs) \{r _(i) ∈R _(obs) |A(r _(i))<η₁ ∧E(r _(i))<η₂}

R ₂ =R ₁ \{r _(i) ∈R ₁|maxline(r _(i))<η₃},

R ₃ =R ₂ \{E(r _(i))<η4∧S(r _(i))<η₅}.

The function l=maxline(r_(i)) returns the length of the longest linesegment found on the boundary of region r_(i). A Split- and-Mergealgorithm can be utilized for detecting lines in region boundaries. Insome embodiments, η₁=0.0, η₂=2.0, η₃=0.7, η₄=0.8, η₅=0.38 were used asexamples. Additionally or alternatively, clutter can be distinguishedresponsive to predetermined movements of the robot over a given timeperiod or surface area. For example, obstacles detected responsive to apredetermined number of robot collision events or non-contact obstacledetection with an optical or acoustic sensor and/or changes in directionin a relatively small area and/or distance of travel may be used todistinguish clutter. In a specific example, numerous detections ofobstacles to the left or right of the robot when traveling in asubstantially straight path over longer distances may be indicative ofwalls or corridors, while numerous detections of obstacles to the leftor right of the robot responsive to non-linear travel over shorterdistances (and accompanying changes in the direction of travel) may beindicative of clutter. Also, a “shape” of an obstacle defined by thecollision events and/or changes in direction of the robot may bedetected and matched with shapes known to correspond to clutter, forinstance, as stored in a data store based on labeling inputs receivedfrom multiple users.

For example, a robot encountering an obstacle or collection of obstaclesconstituting clutter may detect and avoid the clutter by turning from aforward heading at least three times in a two foot distance. Forexample, a robot encountering an obstacle or collection of obstaclesconstituting clutter may detect and avoid the clutter by turning leftand/or right at least three times in a distance of less than 10 feet.For example, a robot attempting to wall follow along an edge of anobstacle or collection of obstacles constituting clutter may detect andavoid the clutter by turning left and/or right at least three times in adistance of less than 5 feet. For example, a robot attempting to wallfollow along an edge of an obstacle or collection of obstaclesconstituting clutter may detect and avoid the clutter by turning leftand/or right at least three times in a distance spanning the length ofthe clutter over a length of 10 feet or less. For example, a robotencountering an obstacle or collection of obstacles constituting cluttermay detect and avoid the clutter by turning from a forward heading atleast three times in a two foot distance. For example, a robotencountering an obstacle or collection of obstacles constituting cluttermay detect and avoid the clutter by turning left and/or right at leastthree times in a distance of less than 10 feet along one primarydirection and at least 10 feet along a direction orthogonal to theprimary direction so as to identify an outer boundary of the clutterregion.

Once the segmentation map is generated, the segmentation map may be sentto a user device that includes a display that is configured to displaythe segmentation map of the surface (block 504). Some embodimentsprovide that the user device includes a wireless communication device400. In some embodiments, the segmentation map may be wirelesslytransmitted using at least one wireless communication link.

Some embodiments provide that the wireless communication link mayinclude a local area network that provides communication between theuser device and the mobile floor cleaning robot. In some embodiments,the user device and the mobile floor cleaning robot may communicate withone another directly via one or more near field communicationsprotocols, such as Bluetooth or Wi-Fi, among others. In someembodiments, the user device and mobile floor cleaning robot maycommunicate with one another through one or more intervening wirelesscommunication devices. In some embodiments, the wireless communicationlink includes a server that is configured to provide a persistentstorage of the occupancy grid, the segmentation map and/or a modifiedsegmentation map, among others.

In some embodiments, the segmentation map includes operational datacorresponding to the mobile floor cleaning robot, such as thesegmentation maps shown in FIGS. 20G-20K. Operational data may include acleaning function map that identifies dirt and/or debris collection thatcorresponds to the quantity of dirt and/or debris that is collected bythe mobile floor cleaning robot as a function of location on thesegmentation map. In some embodiments, the cleaning function map may bea heat map that indicates which portions of the enclosed space were thedirtier than other portions. Some embodiments provide that the cleaningfunction map may be provided for each of the regions individually. Insome embodiments, the cleaning function map may correspond to either asingle cleaning mission or multiple cleaning missions. The multiplecleaning missions may include a defined subset of all cleaning missionscompleted in the enclosed space and/or region or may corresponding toall of the cleaning missions for which data is stored.

Some embodiments include sending the segmentation map of the surface toa remote computing device (block 506). Some embodiments provide that theremote computing device may include electronic device 300 describedabove. In some embodiments, the remote computing device 300 maycommunicate with the mobile floor cleaning robot and with a user mobileterminal, such as wireless communication device 400 described above.Some embodiments provide that the segmentation map is sent to the remotecomputing device 300 in response to the generation of the segmentationmap. In this manner, historical data corresponding to the mobile floorcleaning robot may be preserved, maintained, and/or analyzed.

Some embodiments provide that a modified segmentation map may bereceived from the user device based on inputs received from the userinto the user device (block 508). The modified segmentation map mayidentify a subset of the regions to be cleaned. For example, a userdevice may receive an input instructing that specific regions be cleanedwithout cleaning other ones of the regions, that is, an input specifying“keep-out” regions, as shown for example, in FIG. 27. Additionally, someembodiments provide that the modified segmentation map identifies aspecific order in which the selected regions are to be cleaned, as shownfor example, in FIG. 15. In some embodiments, the modified segmentationmap may include different cleaning schedules for different times and/orintervals. For example, a first cleaning schedule may choose a firstsubset of regions for cleaning on certain days of the week while asecond cleaning schedule may choose a second subset of regions forcleaning on other days of the week. Some embodiments provide that someregions may be included in both the first and the second subset.

Operations also include generating an automated floor coverage pattern(block 510).

Some embodiments provide that the automated floor coverage pattern isbased on the segmentation map that includes different regions, asdiscussed below with reference to FIGS. 29A-29D. In some embodiments,the automated floor coverage pattern includes identification ofsub-regions as clutter areas and non-clutter areas, as shown for examplein FIG. 29A, and selection of cleaning sequence and rank direction foreach of the regions and/or sub-regions, as shown for example in FIGS.30A-30G. Rank direction may be the primary direction that the mobilefloor cleaning robot travels in when cleaning a particular region. Forexample, the rank direction may define a dimension of a given regionthat is longer than other dimensions of the region, as shown for examplein FIG. 31A. By selecting the rank direction, each region may be cleanedwith fewer turns and/or less time.

In some embodiments, the rank direction may define a dimension of asub-region or portion of a region that is without obstacles, that is, anon-clutter area or region, such as the regions 914 that do not includedetected clutter 916 shown in FIG. 9B. For example, in a region thatincludes significant clutter areas, the rank direction may be selectedbased on the dimensionality of the non-cluttered region, regardless ofthe overall dimensions of the region. Some embodiments provide that theautomated floor coverage pattern includes, for each region, a cleaningpattern that includes sequentially selecting an uncluttered portion ofthe region for coverage (FIG. 29B), a perimeter of the region forcoverage (FIG. 29D), and a cluttered portion of the region for coverage(FIG. 29C).

Some embodiments include performing a cleaning operation on the surfaceby sequentially performing the cleaning operation in individual ones ofthe regions and/or sub-regions based on the automated floor coveragepattern (block 512). As such, the cleaning operation in a given regionor sub-region (such as a non-clutter area or areas) is completed beforestarting the cleaning operation in another region (such as a clutterarea or areas), according to the sequence specified by the automatedfloor coverage pattern. Some embodiments provide that the cleaningoperation is further performed based on the modified segmentation map.For example, the cleaning operation may only be performed on a subset ofthe regions or sub-regions responsive to a selection input received fromthe user device.

In some embodiments, the cleaning operation may be performed on specificregions at different respective cleaning levels that may be selected bythe user. Different cleaning levels may correspond to a different numberof cleaning passes that the mobile floor cleaning robot makes in a givenregion. For example, the user may specify that a first region gets ahigher level cleaning operation that corresponds to the mobile floorcleaning robot making three cleaning passes in the region. In contrast,a second region may be specified to have a lower level of cleaning thatcorresponds to a single cleaning pass in the region.

FIG. 6, which is a block diagram illustrating operations of computerreadable program product including computer readable program code 600 ona user device according to some embodiments of the present invention.Some embodiments provide that the user device includes a user mobileterminal, such as wireless communication device 400 described above.Operations include receiving occupancy data and/or a segmentation map ofa surface of an enclosed space from a robot (block 602). As used herein,a segmentation map may refer to a segmented map showing demarcatedregions on a mapped area. The segmentation map may include multipleregions of the surface and the robot may be a mobile floor cleaningrobot.

In some embodiments, the segmentation map is generated by the robotbased on data gathered during exploration and/or cleaning missions onthe surface. Some embodiments provide that the segmentation map isgenerated by a remote computing device that may include electronicdevice 300 as described above. In such cases, the remote computingdevice may receive raw data corresponding to surface of the enclosedspace from the robot and the raw data may be used to generate thesegmentation map.

An image of the segmentation map may be displayed on the display of theuser device (block 604). Some embodiments provide that user device is ahand-held mobile terminal and the display is a touch-screen display. Insuch embodiments, a graphical user interface may be generated anddisplayed via the touch-screen display and the user input may bereceived via the graphical user interface using the touch-screen. Thedisplay may include a monitor, a tablet computer, a networked deviceincluding a screen and a mouse or touch interface, and/or a combinationof remote and local computing devices that are communicatively coupledvia a network.

A user input corresponding to the segmentation map of the surface may bereceived via a user input device of the user device (block 606), asshown for example in FIGS. 10A-10F. Some embodiments provide that theuser input received includes a user input to modify an appearance of theimage of the segmentation map. For example, the user input may be tochange an angle of perspective of the user relative to the image, suchas changing the pan or tilt of the image. The user input may correspondto a dimensional selection such as two-dimensional or three-dimensional.Additionally, a user input may change the zoom to view some or all ofthe image at different sizes.

In some embodiments, the user input may include a map feature selectionsuch as a heat map selection that causes the image of the segmentationmap displayed to illustrate a level of dirt and/or debris collected as afunction of location on the segmentation map, as shown for example inFIG. 24. In this manner, the image may illustrate which portions and/orregions have required more cleaning 2420 relative to other portionsand/or regions 2410. Some embodiments provide that a map feature mayinclude a clutter map selection that causes the image of thesegmentation map displayed to selectively illustrate clutter on thesegmentation map, as shown for example in FIGS. 9B and 9C.

Some embodiments provide that the user input to the segmentation map maybe used to modify the segmentation map and/or regions thereof. Forexample, a user input may be a divide region input that is configured tocause one of the regions to be divided into more than one region, asshown for example in FIGS. 11A-11C. Similarly, a user input may be acombine region input that is configured to cause multiple ones of theregions to be combined into a single region, as shown for example inFIGS. 10A-10F.

Some embodiments provide that the user input to the segmentation map mayinclude a cleaning mission instruction. For example, a cleaning missioninstruction may include a cleaning order instruction that includes anidentification of the order in which the regions are to be cleaned in agiven mission, as shown for example, in FIG. 15. The cleaning orderinstruction may include an identification regarding a subset of theregions that are to be cleaned in the given mission. Embodimentsdirected to cleaning mission instructions and associated user inputs aredescribed in greater detail below with reference to FIGS. 29A-32.

In some embodiments, the user input may include one or more cleaningmission schedules for the enclosed space. For example, a cleaningmission instruction may include a first cleaning mission instructionthat identifies a first selection of the regions to be cleaned and asecond cleaning mission that identifies a second selection of theregions to be cleaned. In this example, the first cleaning missioninstruction may be scheduled to be performed on certain days of the weekand/or times of day, while the second cleaning mission instruction maybe scheduled to be performed on different days and/or at differenttimes.

In some embodiments, the cleaning mission instruction includes differentlevels of cleaning for different regions. For example, a first region orset of regions may be designated to receive a first cleaning level and asecond region or set of regions may be designated to receive a secondcleaning level that is different from the first cleaning level.Different cleaning levels may be accomplished by specifying the numberof times the robot has to traverse the corresponding region, alsoreferred to herein as the number of cleaning passes.

In some embodiments, the cleaning mission instruction includes anidentification of a portion of the enclosed space floor space that isnot to be cleaned by the mobile floor cleaning robot, as shown forexample in FIG. 27. Such designations may be selected by region and/orby portion of the region. In some embodiments, the cleaning missioninstruction may specify that the portion of the enclosed space floorspace that is not to be cleaned is a global instruction and thus ispersistent from mission to mission. However, some embodiments providethat the identification of the portion of the enclosed space floor spacethat is not to be cleaned is a one mission and/or a limited number ofmissions instruction.

Some embodiments provide that the cleaning mission instruction includesa primary cleaning direction for each of the regions, as shown forexample in FIG. 29A. The primary cleaning direction, which may bereferred to as rank, identifies a primary cleaning direction that themobile floor cleaning robot may clean in obstacle free portions of acleaning operation. In some embodiments, rank may be determined for anentire region and/or a portion thereof.

Some embodiments provide that the user input includes a request for userpreference data, which may be displayed responsive thereto. Userpreferences that may be displayed include historical user preferencedata and/or user preference data that corresponds to a single cleaningmission.

In some embodiments, the user input includes a robot team configurationinput that corresponds to multiple different robots. The robot teamconfiguration may include a first cleaning mission instructioncorresponding to a first mobile floor cleaning robot and a secondcleaning mission corresponding to a second mobile floor cleaning robot.In some embodiments, the first and second cleaning missions correspondto the entire segmentation map of the surface of the enclosed space. Forexample, a first cleaning mission may clean a first portion of theregions and a second cleaning mission may clean a second portion of theregions.

However, some embodiments provide that the first and second cleaningmissions correspond to different segmentation maps of different portionsof the enclosed space. For example, in the context of a multileveloccupancy, a first segmentation map may correspond to a first level anda second segmentation map may correspond to a second level.

Some embodiments provide that the first mobile floor cleaning robotperforms a first type of cleaning operation and the second mobile floorcleaning robot performs a second type of cleaning operation. Forexample, the first mobile floor cleaning robot may be a vacuuming robotincluding sophisticated mapping sensors and the second mobile floorcleaning robot may be a mopping robot having less sophisticated sensorsand relying on the mapping created by the vacuuming robot.

Some embodiments include sending a modified segmentation map that ismodified corresponding to the user input to the mobile floor cleaningrobot (block 608). Some embodiments provide that the modifiedsegmentation map is sent to a remote server, such as the remotecomputing device 300, that is configured to provide a persistent storageof the modified segmentation map.

A status communication that is generated by the mobile floor cleaningrobot may be received (block 610). Some embodiments provide that amessage is displayed in the on the display responsive thereto.

In some embodiments, the status communication includes a progress updatecorresponding to a cleaning mission. The progress update may bedisplayed as an animated graphic of a cleaning operation on the image ofthe segmentation map. Some embodiments provide that the progress updateis updated in real time and includes a visual indication on the image ofthe segmentation map that is displayed, as shown for example in FIG. 13.

Some embodiments provide that the status communication includes an alertmessage that identifies a condition that is detected by the mobile floorcleaning robot. For example, the alert message may be reporting anincomplete mission condition that renders the robot unable to complete acleaning mission. For example, the mobile floor cleaning robot mayindicate that it is stuck and unable to proceed, as shown for example inFIG. 14. The condition may correspond to an unexpected obstaclecondition. In some embodiments, the mobile floor cleaning robot maydetect an obstacle that is not included in the segmentation map andreport that this is a new obstacle.

FIG. 7 is a block diagram illustrating operations of computer programproduct 700 on a robot management node according to some embodiments ofthe present invention. The robot management node may be implemented onand/or include electronic device 300 as described above. Someembodiments provide that the processer and the memory are included in acloud-based server that is remote from the robot and the enclosed space.

In some embodiments, a cloud-based server may aggregate datacorresponding to multiple different users at multiple different enclosedspaces. Examples of the aggregated data may include data correspondingto standard features, fixtures, floorplans, footprints and/orarrangements. For example, multiple users may share a floor plan that isthe same as other users and that may be identified by name, numberand/or specific dimensional features. Having the data collectedcorresponding to the same or similar features, fixtures, floorplans,footprints and/or arrangements may provide a data repository thataccessible for other users to improve performance and/or accuracyregarding operations such as classifying obstacles.

Operations may include receiving occupancy data and/or a segmentationmap of a surface of an enclosed space from a robot (block 702). Thesegmentation map may identify multiple regions of the surface basedoccupancy data that is collected by the robot. Some embodiments providethat multiple segmentation maps that are generated by the robot duringmultiple cleaning missions are received. In such embodiments, thesegmentation map may be updated and/or augmented by later receivedsegmentation maps to provide persistent maps. Some embodiments providethat the segmentation map includes operational data corresponding to therobot.

The segmentation map of the surface of the enclosed space may be storedin a data store that is associated with the robot (block 704). Someembodiments include storing multiple segmentation maps that arereceived, and transferring common features between the multiplesegmentation maps using feature matching and outlier removal operationsdescribed herein to generate persistent maps. In this manner, historicalsegmentation map data including the modified segmentation map may bestored.

User input data is received from a user device (block 706). The userdevice is configured to receive a user input corresponding to thesegmentation map. Responsive to receiving the user input, thesegmentation map is modified based on the user input data to generate amodified segmentation map. The modified segmentation map may identify asubset of the regions that are to be cleaned, portions or sub-regions ofone or more of the regions to be cleaned, and/or an order in which theregions and/or sub-regions are to be cleaned. The modified segmentationmap may also be stored and re-used in subsequent navigation and/orcleaning operations performed by the mobile robot.

In some embodiments, the segmentation map includes operational datacorresponding to the robot. Operational data may include a cleaningfunction map that identifies dirt and/or debris that was collected in asingle cleaning mission and/or in multiple cleaning missions as afunction of location on the segmentation map. In this manner, a heat mapthat identifies regions and/or portions thereof that require morecleaning activity relative to other regions and/or portions thereof maybe communicated. For example, some embodiments provide that theoperational data includes a cleaning function map that identifies aquantity of passes that are performed by the robot as a function oflocation on the segmentation map.

In some embodiments, the user input data includes a robot teamconfiguration input that includes a first cleaning mission instructioncorresponding to a first robot and a second cleaning missioncorresponding to a second robot.

Operations include sending the modified segmentation map based on theuser input data and the segmentation map to the robot (block 708). Therobot may be operable to perform a cleaning mission responsive to themodified segmentation map.

In some embodiments, the cleaning mission includes performing a cleaningoperation of regions at different cleaning levels that are selected bythe user. The cleaning mission may further provide a rank direction thatcorresponds to a largest dimension of area in the respective regionwithout obstacles. The different levels may be associated with a numberof cleaning passes the robot makes in the corresponding regions. In someembodiments, the rank direction defines a dimension of the region thatis longer than other dimensions of the region. Some embodiments providethat the rank direction defines a dimension of a portion of therespective region that is without obstacles and that is different from adimension of the region that is longer than other dimensions of theregion.

FIGS. 8A through 8C are graphical representations that may be displayedvia a user interface of a communication device illustrating a raw datamap, a cleaned map and a segmentation map in an automated map cleaningand segmentation operation according to some embodiments of the presentinvention. Some embodiments provide that a robot explores a surface ofan enclosed space during a first pass. An occupancy grid is generatedthat includes raw data that is generated during the first pass. Theoccupancy grid is presented to a user on a user device as a raw data map802 as illustrated in FIG. 8A. The raw data map 802 may include multiplepixels that may be shaded white for traversable space and black forobstacles, including occupancy walls.

Referring to the embodiment of FIG. 8B, the robot generates a cleanedmap 812 that is displayed to the user on the user device. The cleanedmap 812 may illustrates the enclosed space with the boundary data 816cleaned up or simplified to be substantially more linear than the rawdata. The cleaned map 812 also highlights the traversable space 814 ofthe enclosed space surface, for example, by removing visual indicationsof the detected clutter.

FIG. 8C illustrates a segmentation map 822 that may be displayed to theuser on the user device. Similar to the cleaned map 812, thesegmentation map 822 illustrates the enclosed space with the boundarydata 826. Additionally, the traversable space is segmented into regions824 that may correspond to different rooms in the enclosed spaceaccording to some embodiments. Based on the segmentation map 822, therobot may perform room-by-room cleaning in which the cleaning operationin a given region 824 is completed before the robot begins to perform acleaning operation in another region 824.

FIGS. 33A through 33D illustrate operations for generating simplifiedmap boundaries for display via a user interface from raw data maps, suchas the map 802 in FIG. 8A, in greater detail. Referring now to FIG. 33A,a raw data map 3302 is generated from data collected during navigationof a mobile robot an operating environment, such as occupancy data andVSLAM feature detection data. The raw data map 3302 provides a pixelatedrepresentation of the operating environment, which may not beaesthetically pleasing and/or intuitive to a user. As such, the raw datamap 3302 can be visually simplified for presentation to a user byremoving noise to generate the de-noised map 3312 shown in FIG. 33B, andremoving areas identified as clutter to generate a de-cluttered map 3322as shown in FIG. 33C. The de-noised map 3312 may have reduced distortionrelative to the raw data map 3302, for example, by sharpening orreducing blurred edges. Pixelated edges of the decluttered map 3322 arereplaced by linear segments and/or polygonal shapes to generate thesimplified map 3332 shown in FIG. 33D. The simplified map 3332 thusreplaces data indicating actual boundaries detected by navigation of themobile robot with simplified boundaries, which may be more aestheticallypleasing and/or intuitive for a user viewing the map 3332 via a userinterface of a user device, such as the user device 202. Generating thesimplified map 3332 may also involve merging rooms or regions asdescribed herein. In some embodiments, clutter regions 3306 representingthe areas identified as clutter in FIG. 33B may be added back into tothe simplified map 3332 to provide a more accurate representation of theoperating environment to the user. Also, because clutter obstacles canrepresent recognizable structures, such as furniture, the inclusion ofthe clutter regions 3306 in the simplified map 3332 may make it easierfor the user to recognize rooms and corridors. The operations togenerate the simplified map 3332 from the raw data map 3302 may beperformed by the robot 100, the robot management server 200, and/or theuser device 202.

Some embodiments provide that a user may have a display preferencerelative to how clutter regions may be displayed. FIGS. 9A through 9Care graphical representations that may be displayed via a user interfaceof a communication device illustrating user display preference optionsof raw data, a segmentation map with clutter pixels and segmentation mapwith clutter regions in operations according to some embodiments of thepresent invention. Referring to FIG. 9A, a raw data map 902 may includethe identification of multiple pixels that correspond to clutter 906 orother obstacles in the enclosed space. As shown in FIG. 9B, asegmentation map 912 may include multiple regions 914 and the clutter916 or other obstacles may be illustrated in each of the regions 914. Insome embodiments, as illustrated in FIG. 9C, a segmentation map 922 mayinclude clutter regions 926 within the regions 924 that illustratespecific areas in which multiple obstacles are detected. As describedherein, clutter may refer to a concentration of obstacles occupying aregion of space and preventing systematic, deterministic coverage of theregion, and regions including clutter may be referred to herein asclutter or cluttered regions. As described in greater detail below, oncethe cluttered regions are identified, cleaning operations based on rankoptimization may be performed on non-cluttered regions.

FIGS. 10A through 10F are graphical representations 1002 that may bedisplayed via a user interface of a communication device illustratingoperations for performing user region merging on a segmentation mapaccording to some embodiments of the present invention.

Referring to FIG. 10A, a user may select a particular region 1005 a ofthe displayed image of the segmentation map. The selection may beperformed using a pointing input such as a mouse, trackball and/orstylus and/or via a touch screen user input. FIG. 10B illustrates thedisplayed image of the segmentation map with the selected region 1005 ahighlighted.

Referring to FIG. 10C, the user may select another region 1005 c of thedisplayed image of the segmentation map that is adjacent the firstselected region. FIG. 10D illustrates the displayed image of thesegmentation map with both of the selected regions 1005 a, 1005 chighlighted. Referring to FIG. 10E, the user may provide a user input tomerge the selected regions 1005 a, 1005 c into a single region 1005 e.The user input may be received via a menu that identifies actions and/orby a voice command. FIG. 10F illustrates the displayed image of thesegmentation map with multiple of the original regions merged into asingle region 1005 f. In some embodiments, the merge operation may beperformed multiple times on two regions to generate a single region thatincludes more than two of the originally defined regions. Someembodiments provide that more than two of the regions may be selectedbefore the user input to perform the merge operation is received.

FIGS. 11A through 11C are graphical representations 1102 that may bedisplayed via a user interface of a communication device illustratingoperations for performing user region splitting on a segmentation mapaccording to some embodiments of the present invention. In someembodiments, for various reasons, a user may determine that a region asprovided in the segmentation map should be divided into more than oneregion. For example, a single space may have a defined kitchen area andadjoining living space separated by various design artifacts, such asflooring type changes, and may be treated differently based on cleaningneed. Referring to FIG. 11A, an endpoint 1105 a of a division line inthe region to be divided may be selected by the user. Then, referring toFIG. 11B, the other endpoint 1105 b of the division line in the regionto be divided may be selected by the user. The user may thus provide auser input to divide the selected region into the two regions 1105 c 1,1105 c 2 shown in FIG. 11C. In some embodiments, the division operationis selected before the endpoints 1105 a, 1105 b of the division line areidentified. In such embodiments, the division operation mayautomatically be performed once the second endpoint 1105 b of thedivision line is selected.

FIGS. 12A through 12D are graphical representations 1202 that may bedisplayed via a user interface of a communication device illustratingoperations for performing user region labeling on a segmentation mapaccording to some embodiments of the present invention. Each of theregions of the segmentation map may be labeled according to a userpreference. Referring to FIG. 12A, a labeling operation may begin withthe user selecting a region 1205 a of the segmentation map. FIG. 12Billustrates text box 1205 b that may be displayed that includes thelabel provided by the user. For example, in the current example, theuser selects a region in FIG. 12A and then inputs the label “DEN” 1205 bfor the selected region 1205 a in FIG. 12B. FIGS. 12C and 12D illustratesimilar operations corresponding to the selection 1205 c and labeling1205 d of the “LIVING ROOM” region.

FIG. 13 is a graphical representation 1300 that may be displayed via auser interface of a communication device illustrating operations fordisplaying progress of a robot on a segmentation map according to someembodiments of the present invention. As illustrated, the display of thesegmentation map may include an indication of the cleaning progresscorresponding to a cleaning mission being performed. For example,cleaned regions and/or portions thereof 1302 may be represented by afirst color, pattern, and/or shading that is different from the color,pattern and/or shading representing uncleaned regions and/or portionsthereof 1304. In this manner, a user may be informed as to which regionsand/or portions thereof may expect or not expect to be occupied by therobot.

FIG. 14 is a graphical representation 1402 that may be displayed via auser interface of a communication device illustrating operations forproviding an alert 1405 to a user of a failure condition on asegmentation map according to some embodiments of the present invention.In some embodiments, the display may include a message and/or othersymbol that is generated based on a message from the robot. For example,the communication device may receive a message that is generated by therobot that indicates a condition that has or is occurring during acleaning mission. For example, if the robot gets stuck or trapped, analert message may be received and displayed to the user. Other examplesmay include alerts generated by the robot that indicate that the robotmay not have completed a cleaning mission.

FIG. 15 is a graphical representation 1502 that may be displayed via auser interface of a communication device illustrating operations forreceiving user input to provide a cleaning sequence (1, 2, 3) of a roboton a segmentation map according to some embodiments of the presentinvention. In some embodiments, a user may provide a user inputcorresponding to a cleaning order. For example, each of the regions ofthe segmentation map may be labeled with a cleaning order number 1, 2,3. In some embodiments, the cleaning order may be entered via a table1505 of the regions that is displayed to the user, while otherembodiments provide that the user simply selects the region and entersthe value corresponding to the order of cleaning. Some embodimentsprovide that some of the regions may be deselected for cleaning during agiven mission. For example, as illustrated, the kitchen and living roommay have an order number of “X” that indicates that these regions arenot to be cleaned.

FIGS. 16A through 16G are graphical representations 1602 a-1602 g thatmay be displayed via a user interface of a communication deviceillustrating operations for map data aggregation using multiple runs inthe same occupancy according to some embodiments of the presentinvention. FIGS. 16A-16D illustrate displays of a raw data maps thatinclude additional raw data that is gathered by the robot duringsubsequent runs in the same occupancy. For example, for FIG. 16A to FIG.16B the raw data map undergoes autorotation (180 degrees) and alignment.As illustrated, the raw data from FIG. 16B reflects an alignment ofdifferent portions of the raw data relative to FIG. 16A and a 180 degreerotation relative to the initial raw data illustrated in FIG. 16A. FIGS.16C and 16D illustrate that additional raw data is used to substantiallycomplete the raw data map. For example, FIG. 16C illustrates additionalraw data being added relative to FIG. 16B, thus advancing the overallcompletion of the raw data map. Similarly, FIG. 16D illustrates evenfurther map completion of the raw data. FIGS. 16E through 16G illustratethe removal of non-static obstacles detected intermittently or only onone or some of the total missions. In this manner, the integrity,accuracy and/or content of the raw data map may be increased during eachadditional run. For example, FIG. 16E illustrates clutter detection in aportion of a room. FIG. 16F illustrates identification of non-staticobjects that are part of the detected clutter and FIG. 16G illustratesthe removal of non-static objects from the raw data map. It will beunderstood that persistency of the map may be required in order toaccurately overlay maps from each run shown in FIGS. 16A-16D with eachother. Also, clutter detection can be significantly improved because,when observing obstacles over time, the location of static obstacles maystay the same whereas the locations of clutter/non-static obstacles maychange over time.

FIGS. 17A and 17B are graphical representations 1702 a and 1702 b thatmay be displayed via a user interface of a communication deviceillustrating a segmentation map from the aggregating operations based onthe raw data maps illustrated in FIGS. 16A-16G that are without clutterand that include clutter, respectively, according to some embodiments ofthe present invention. As illustrated, FIG. 17A is an image of asegmentation map 1702 a corresponding to the raw data maps illustratedin FIGS. 16A-16G without clutter. In contrast, FIG. 17B is an image of asegmentation map 1702 b corresponding to the raw data maps illustratedin FIGS. 16A-16G with clutter regions identified in the correspondingsegmentation map regions.

FIGS. 18A and 18B are graphical representations 1802 a and 1802 b thatmay be displayed via a user interface of a communication deviceillustrating raw data of an occupancy map and a three-dimensionalrepresentation of the occupancy map, respectively, based on the raw datamaps illustrated in FIGS. 16A-16G according to some embodiments of thepresent invention. As illustrated, FIG. 18A is the raw data map 1802 aillustrated above as FIG. 16E and FIG. 18B is an image of athree-dimensional representation 1802 b of the occupancy map basedthereon.

FIG. 19 is a graphical representation 1900 that may be displayed via auser interface of a communication device illustrating an occupancy mapillustrating operations for generating persistent informationcorresponding to an enclosed space according to some embodiments of thepresent invention. The occupancy map 1900 illustrates the floor plan ofthe enclosed space as mapped by the robot. The robot cleaning path 1910is identified over all of the available floor surface of the enclosedspace. Additionally, landmarks 1904, which may be determined, generatedand/or assigned by the robot, are identified throughout the occupancymap 1900. Specific regions corresponding to multiple ones of thelandmarks 1904 may be identified as landmark regions 1902 and mayinclude trajectories and/or primitives defined by the landmarks 1904. Insome embodiments, the term “landmark” may include locations that arespecifically identified in one or more maps and may also be referred toas waypoints.

FIGS. 20A through 20K are graphical representations 2002 a-2002 k thatmay be displayed via a user interface of a communication deviceillustrating views of occupancy maps and segmentation maps correspondingto data gathered during robot exploration of an enclosed space accordingto some embodiments of the present invention. As illustrated, FIG. 20Aincludes an image of a raw data occupancy map 2002 a that includesclutter and other non-traversable regions within the traversable floorspace and the path that the robot traversed during the explorationand/or cleaning pass of a cleaning mission. A raw data map 2002 b of theenclosed space is provided in FIG. 20B and a clean occupancy map 2002 cis illustrated in FIG. 20C. FIG. 20D illustrates a display of a cleanoccupancy map 2002 d with clutter pixels 2008 d. A clean map 2002 eincluding the path 2005 e that the robot traversed without clutterdisplayed is illustrated in FIG. 20E and a clean map including the path2005 e that the robot traversed with clutter displayed as solidly filledpatches 2008 f is illustrated in FIG. 20F.

A display of a segmentation map 2002 g of the enclosed space withoutclutter is illustrated in FIG. 20G and a display of the segmentation map2002 h of the enclosed space with clutter displayed as solidly filledpatches 2008 h is illustrated in FIG. 20H. FIG. 20I illustrates adisplay of a segmentation map 2002 i including the path 2005 e that therobot traversed without the clutter displayed and FIG. 203 illustrates adisplay of a segmentation map 2002 j including the path 2005 e that therobot traversed with the clutter 2008 j displayed. A display of thesegmentation map 2002 k that includes boundaries between regions thereinand that includes region numbers 1-9 is illustrated in FIG. 20K.

FIG. 21 is a diagram illustrating a system of devices that may be usedto provide robot management according to some embodiments of the presentinvention. The system may include a robot 2102 that may generateoccupancy data and/or a segmentation map 2110 of a floor surface of anenclosed space. The robot 2102 may be the same as the robot 100described herein. As such, additional description thereof will beomitted. In some embodiments, the occupancy data and/or segmentation map2110 may be transmitted to a cloud device and/or system 2104. In someembodiments, the cloud device and/or system 2104 may include the robotmanagement node 200 described herein. The cloud device and/or system2104 may include a processor 2105 and user data store 2106 that maystore user input history (including identifications of clutter patternsfrom a plurality of users), occupancy map history, segmentation maphistory, and/or robot performance and/or communication history, whichmay be used in accordance with embodiments described herein to maintainand update persistent maps. While some embodiments provide that therobot 2102 performs the computation of the segmentation map 2110, insome embodiments the computation may be performed by the cloud deviceand/or system 2104 based on the occupancy data collected by the robot2102.

In some embodiments, the cloud device and/or system 2104 may provide acommunication channel between the robot 2102 and a user device 2120. Theuser device 2120 may be the same as the user device 202 describedherein. The user device 2120 may provide user inputs that allow a userto edit and/or label an occupancy and/or segmentation map. Further, theuser device 2120 may be used to modify the segmentation map to includespecific instructions for one or more cleaning missions. Suchinstructions may include cleaning order, rank direction, and/or cleaninglevel, among others. The user device 2102 may also provide inputs forselecting display characteristics such as clutter inclusion/exclusion,view dimensionality, zoom, viewing angle, cleaning mission playbackoptions, cleaning progress, expected time of cleaning missioncompletion, historical, per use and/or aggregate data, among others.

Additionally, the user device 2120 may be operative to receive messagesand/or data from the robot 2102 and/or the cloud device and/or system2104. For example, the user device 2120 may receive performance datacorresponding to the status and/or performance of the robot 2102 and/oralert messages that are generated by the robot 2102, among others. Forexample, a heat map that displays the level of cleaning that wasperformed and/or the amount of dirt and/or debris that was collected asa function of location may be provided.

FIG. 22 is a graphical representation that may be displayed via a userinterface of a communication device illustrating history screen 2200including a mission coverage map 2202 corresponding to a cleaningmission according to some embodiments of the present invention. Asillustrated, responsive to receiving a user input to provide cleaninghistory, the user device may display a history screen 2200 that includestime information such as the date 2210, a mission coverage map 2202, andcleaning mission performance data 2203. Cleaning mission performancedata 2203 may include the duration of the cleaning mission, the startand/or end time, a mission status, the amount of area cleaned during thecleaning mission and/or a dirt detection count.

FIG. 23 is a graphical representation that may be displayed via a userinterface of a communication device illustrating a map 2302 includingreal-time reporting corresponding to a cleaning mission of an enclosedspace according to some embodiments of the present invention. Responsiveto receiving a user input requesting cleaning mission progress data, theuser device may display the occupancy map and/or segmentation map 2302including an identification of which regions and/or portions thereof arecleaned 2305 and which regions and/or portions thereof that are notcleaned 2307 in the current cleaning mission. Some embodiments providethat additional text 2303 may be provided that identifies which regionshave been cleaned and which region is in the cleaning process.

FIG. 24 is a graphical representation that may be displayed via a userinterface of a communication device illustrating a map including amulti-mission coverage display according to some embodiments of thepresent invention. As illustrated, the occupancy and/or segmentation mapmay be displayed to include historical coverage data corresponding tomultiple cleaning missions. For example, a heat map 2402 may begenerated that identifies areas of the enclosed space that have beenhistorically dirtier relative to other areas. The heat map 2402 mayinclude different colors, shading, and/or brightness/intensity toillustrate a level of dirt and/or debris collected at respectivelocations. For example, the shading in the lower right area 2420indicates a greater amount of detected and/or collected debris and/ordirt relative to the upper left area 2410, which is not shaded.

FIG. 25 is a graphical representation that may be displayed via a userinterface of a communication device illustrating room-based data 2503that may be displayed according to some embodiments of the presentinvention. In some embodiments, the user device may include a user inputfor display of room based statistics 2503. For example, some embodimentsprovide that a user may select a region 2501 on a segmentation map 2502and that data 2503 corresponding to that region 2501 may be displayedresponsive to receiving the input. For example, in response to a userinput selecting the living room 2501 on the segmentation map 2502, theuser device may display summary cleaning mission data 2503 correspondingto the living room including how many times the living room was cleaned,the date of the last cleaning, the total time spent cleaning the livingroom, and/or a date range corresponding to the displayed data.

FIG. 26 is a graphical representation demonstrating operations in whicha user input may be provided via a communication device using a voicecommand 2601 according to some embodiments of the present invention. Insome embodiments, the user device 2600 may include user inputs to cleana specific room. For example, the user input may be to clean a specificroom immediately. In some embodiments, the user input may be receivedvia voice command 2601 using voice recognition technology. Someembodiments provide that the voice command 2601 may be received from adevice that is external to the user device 2600. For example, householdmanagement devices that may receive voice commands to perform otherfunctions such as HVAC control, entertainment system control, etc. mayreceive the voice command 2601 and translate the message to the userdevice 2600, the robot 100, and/or the robot management node 200.

FIG. 27 is a graphical representation of receiving a user input todefine a cleaning mission boundary 2705 responsive to a gesture on atouch screen of a communication device according to some embodiments ofthe present invention. In some embodiments, a condition in a particularroom of the enclosed space 2702 may be utilized in a manner that rendersthe cleaning operation undesirable. In such circumstances, a boundary2705 may be added to a region of the segmentation map via a user inputof the user device. For example, a simple gesture on a touch screenblocking a region 2707 may cause the modified segmentation map toexclude that region 2707 from subsequent cleaning missions.

FIGS. 28A through 28C are graphical representations that may bedisplayed via a user interface of a communication device illustrating araw data map 2802, a cleaned map 2812, and a cleaned map 2822 withdetected clutter regions 2824 according to some embodiments of thepresent invention. As discussed above regarding FIG. 8A, someembodiments provide that a robot may explore a surface of an enclosedspace during a first pass. An occupancy grid may be generated thatincludes raw data that is generated during the first pass. The occupancygrid may be presented to a user on a user device as a raw data map 2802as illustrated in FIG. 28A. The raw data map 2802 may include multiplepixels that may be shaded white for traversable space and black forobstacles, including occupancy walls.

Referring to FIG. 28B, the robot may generate a cleaned map 2812 thatmay be displayed to the user on the user device. The cleaned map 2812may illustrate the enclosed space with the boundary data 2816 cleaned upto be substantially more linear than the raw data and may highlight thetraversable space 2814 of the enclosed space surface.

FIG. 28C illustrates cleaned map with detected clutter regions 2822 thatmay be displayed to the user on the user device. Relative to the cleanedmap 2812, the cleaned map with detected clutter regions 2822 illustratesthe enclosed space with the clutter regions 2824. In some embodiments,the operations may further determine whether the clutter 2824 is dynamicclutter or static clutter. For example, the robot may determine thatclutter 2824 is static clutter by determining a probability that theclutter will be encountered more frequently, and/or by accessing a datastore including patterns of clutter identified by one or more otherrobots and/or users, as described below with reference to FIGS. 35Athrough 35D. Some embodiments provide that clutter that is determined tobe static may be marked on the map using darker cells relative toclutter that has a lower probability of being encountered repeatedly.The darker cells may indicate known areas that are not traversable. Byidentifying such features, the robot may be operative to identifyclutter regions that may be cleaned after traversable space is cleaneddeterministically in ranks. Cleaning non-clutter regions first canincrease mission efficiency.

FIGS. 29A through 29D are graphical representations that may bedisplayed via a user interface of a communication device illustrating asegmentation map that includes respective area-specific cleaningoperations according to some embodiments of the present invention.Referring to FIG. 29A, a segmentation map 2902 includes regions andportions thereof identified as open area 2904 and clutter area 2906. Asprovided herein, an open area may refer to an identified area of theenclosed space, regions and/or portion thereof that is identified ashaving few or no identified clutter or clutter areas, and is alsoreferred to herein as a non-clutter area. As mentioned, clutter mayrefer to obstacles occupying a surface and preventing systematic anddeterministic coverage of the surface in ranks (rows). In someembodiments, an entire region of a segmented map may be determined to bean open area or a clutter area. Some embodiments provide that an openarea and/or a clutter area may be defined as a portion of a region, alsoreferred to herein as a sub-region. For example, the clutter area 2906is a portion of a region that also includes defined open areas 2904.Also, an open area 2904 may include small areas of clutter therein. Asshown in FIG. 29A, the displayed segmentation map 2902 may visuallyhighlight these differently-classified sub-regions 2904 and 2906, forexample, by displaying visually-distinct boundaries around the clutterarea 2906 and the non-clutter areas 2904.

In some embodiments, the clutter area may be determined based on aclutter density determination. For example, some embodiments providethat clutter density may be determined by measuring dimensions betweenspecific clutter points and nearby obstacles or clutter points. Thisdimension and/or function thereof may be compared to some dimension ofthe robot and/or portion thereof. A result of the comparison maydetermine whether or not to classify the obstacle as clutter. Forexample, if the measured distance is less than a threshold that is basedon a dimension of the robot, then the obstacle may be determined to beclutter. Relative dimensions of the robot may include total robotlength, width, and/or cleaning head work width, among others.Non-limiting examples include a range of about 70% to about 300% of thework width of the cleaning head. In other words, obstacles that arespaced apart by a distance of about 70% to about 300% of the width ofthe cleaning head or roller(s) of the robot may constitute (or beinterpreted as) “clutter” because the robot cannot traverse floor spaceoccupied by such obstacles in unimpeded, deterministic ranks. In theexample of FIG. 29A, the lowermost area shown is classified as an openor non-clutter area 2904 based on the relatively low density of clutter2908 therein.

Once the open areas 2904 and/or the clutter areas 2906 are identified, aroute and strategy for cleaning the different areas of the surface ofthe enclosed space, also referred to herein as a coverage pattern, maybe determined by the mobile robot 100, the robot management node 200,and/or the user device 202. The coverage pattern may define an improvedor optimized cleaning strategy that treats areas differently withrespect to their recognized context or classification, for instance, byindicating sequential cleaning of regions or sub-regions of the surfacebased on their respective classifications. For example, the coveragepattern may indicate first executing cleaning of the open areas 2904 asshown in FIG. 29B so as to temporarily ignore and avoid the clutterareas 2906, then executing cleaning of the clutter areas 2906 as shownin FIG. 29C, and then executing cleaning of the perimeter 2930 as shownin FIG. 29D.

Referring to FIG. 29B, in performing the open area cleaning, a rankdirection 2910 may be determined for each of the open areas 2904. Insome embodiments, the rank direction 2910 may correspond to the longestdimension in the given region. However, some embodiments provide thatthe rank direction is based on the dimensionality of the open area thatmay be different from that of the region, for example, as specified by auser input. Once the cleaning operation of the open areas is determined,then an open area cleaning operation may be performed. This improvesefficiency in some embodiments because the robot is not slowed bynavigation of clutter (and associated battery depletion) in areas 2904.As the open areas 2904 can represent a majority of the floor space to becleaned, the robot can complete cleaning of a majority (for example, upto 80% to 90% or more) of the floor space first, and then focus on themore laborious and potentially mission-disrupting clutter areas 2906.For example, the robot could be trapped between chairs or otherfurniture in cleaning the clutter areas 2906 in some instances, therebydisrupting the cleaning mission with respect to open areas 2904.

The open area cleaning may include determining advantageous locationsfor ending the cleaning operation in a particular region or portionthereof. For example, a cleaning operation for a first space that endsproximate a planned starting point for a next region to be cleaned mayreduce the unnecessary traversal across a region or portion thereof thathas already been cleaned. In some embodiments, the next region to becleaned may be determined based on its proximity to the ending point ofthe previous cleaning operation. For example, the room adjacent theclosest doorway at the completion of the current cleaning operation maybe selected to be the next region and/or portion thereof to be cleaned.

In the examples of FIGS. 29A through 29D, the coverage pattern indicatessequential navigation of differently-classified areas, that is, once theopen area cleaning operation is performed, the clutter area cleaningoperation is performed. For example, FIG. 29C illustrates the remaininguncleaned areas are the clutter areas 2906 and the cleaning path forcleaning those areas (shown by arrows indicating a random pattern ofmovement within the clutter areas 2906). As illustrated in FIG. 29D,once the open areas and clutter areas are cleaned, a perimeter cleaningoperation may be performed along the perimeter 2930 of the enclosedspace.

It will be understood that the coverage pattern may not be determinedand/or modified upon initial detection of the clutter areas 2906, aslearning and generation of the map 2902 may require that at least onenavigation or cleaning operation has been performed. That is, incomputing the coverage patterns as described herein, clutter detectionmay be based on several navigations of the surface using persistent mapdata. For example, detected obstacles may be stored in a data store,such as the data store 2106 in the cloud 2104, after each navigation. Bycomparing obstacles and/or structures from several maps generated overtime, probabilities may be computed for each region of the surface,denoting the degree or density of static versus dynamic obstacles (suchas walls versus clutter) therein, which may be used to classify theregions as clutter areas 2906 or open areas 2904. This information canalso be displayed via a user interface of the user device 202, so thatthe classification of the regions as clutter areas 2906 or open areas2904 is observable by a user.

More particularly, after multiple navigations of the surface, the robot100 may be better informed about regions in which the locations ofobstacles may change over time, and such regions can therefore beclassified as clutter areas 2906. Other regions may be determined asbeing static, such as either static free or static occupied, and may beclassified as open areas 2904. Based on this information, a coveragepattern or cleaning strategy can be computed. For example, the coveragepattern may specify cleaning of the open areas 2904 more efficiently ina ranking pattern (as shown in FIG. 29B), and cleaning the clutter areas2906 in an edge cleaning pattern (as shown in FIG. 29C). These examplecleaning behaviors shown in FIGS. 29B and 29C in response to thedetermined coverage pattern can be observed and distinguished. In aparticular example, the robot 100 may first clean efficiently all staticfree areas 2904 by ranking without entering clutter areas 2906. Aftercompleting cleaning of the open areas 2904, the robot 100 may performthe edge cleaning behavior by traversing all cluttered areas 2906sequentially. Also, in some embodiments, the robot 100 may repeatedlyexecute the same pattern during subsequent navigations of the surface.However, when the environment is changed (for example, when clutter isremoved from one or more of the clutter areas 2906), the computedcoverage pattern may be modified by re-classifying one or more of theclutter areas 2906 as an open area 2904, resulting in a change in thebehavior of the robot 100 in navigation of the same surface after there-classification.

For example, in FIG. 29A, elongated open area 2904 may represent theentrance corridor of a house. The majority of such a corridor may befree, so that people can walk when entering or leaving the house. Thelowermost clutter area 2906 to the left of the elongated open area 2904may be a place where people leave their shoes when entering the house.In this area 2906, the robot 100 may therefore frequently bump intothese obstacles (shoes) or otherwise detect the obstacles via anon-contact sensor (for example, a PixArt sensor). In addition, thearrangement of the shoes will typically differ each time the robot 100cleans the area 2906, because the placement of the shoes may frequentlychange over time. Consequently, the robot 100 can learn and classifythis particular area 2906 as dynamically changing (and thus, cluttered),whereas portions of the elongated corridor 2904 (such as the middle andlower areas of the corridor) can be learned and classified as staticfree (such as traversable, open floor space).

Accordingly, a coverage pattern may be determined and provided to therobot 100 such that the robot 100 first cleans the elongated corridor2904 by efficient ranking without entering the shoe-cluttered area 2906.After executing the cleaning of the elongated corridor 2904, the robot100 will sequentially clean the clutter area 2906 (to the left of thecorridor) by edge cleaning, according to the coverage pattern. However,if people subsequently begin placing shoes near the right wall of theelongated corridor 2904 (and no longer in the area 2906 to the left),the area classifications and coverage pattern may be modified responsiveto detection of this change after subsequent navigations, and the robot100 may observably alter its cleaning behavior after a few cleaningruns. More particularly, the coverage pattern may be modified such that,in response to the coverage pattern, the robot 100 may first clean thenow-open area (formerly classified as clutter area 2906) on the leftside of the elongated corridor 2904, and then execute edge cleaning onthe shoe-cluttered area along the right wall of the elongated corridor2904. More generally, there may be multiple indications of clutterdetection during navigation of the robot 100. These may include, but arenot limited to, display and marking of the clutter areas 2906 and openareas 2904 via the user interface of a user device 202, differentbehavior of the robot 100 in navigating the open areas 2904 and clutterareas 2906, and repeated execution of the different behaviors in thesame order or sequence (as well as changes to the repeated executionresponsive to changes in the environment).

In some embodiments, the computing of the coverage pattern and thecleaning of the identified areas of the surface by the mobile robot 100may be responsive to receiving a selection of regions/sub-regions to becleaned, a cleaning sequence/priority, cleaning patterns, and/or a levelof cleaning from a user device, for example, via a user interface 410 ofthe wireless communication device 400 or other user device 202. FIGS.30A through 30G are graphical representations of segmentation map views3002 that may be displayed via a user interface of a user deviceillustrating operations for user selection of respective area-specificcleaning operations and/or preferences to define the coverage patternaccording to some embodiments of the present invention. For example, thesegmentation maps 3002 of FIGS. 30A-30G may visually highlight thedifferently-classified regions to allow user selection of initialcleaning of the open areas 2904 and the sequence in which open areas2904 a, 2904 b, 2904 c, and 2904 d are to be cleaned (illustrated inFIGS. 30A-30D by numerals 1-4), subsequent cleaning of the clutter areas2906 and the sequence in which clutter areas 2906 a and 2906 b are to becleaned (illustrated in FIGS. 30E-30F by numerals 5-6), and completionof cleaning by performing perimeter coverage of the perimeter 2930 ofFIG. 29D (illustrated in FIG. 30G by numeral 7). In performinguser-defined area cleaning operations based on the coverage pattern, themobile robot 100 may treat areas selected for subsequent cleaning as akeep-out zone, until cleaning of the current area is completed. In someembodiments, the mobile robot 100 may traverse thedifferently-classified sub-regions 2904 a-d in substantially linear rankdirections corresponding to a longer dimension of the regions orsub-regions, and may further traverse the sub-regions 2906 in arcs orother non-linear patterns so as to navigate systematically aroundcorners.

Such user-defined area cleaning operations may also allow for userselection of cleaning patterns (including the direction of travel)and/or cleaning level (including the number of passes) performed by themobile robot 100 in executing the cleaning operations, thereby allowingfor user-assigned cleaning levels. FIGS. 31A through 31C are graphicalrepresentations of segmentation map 3102 that may be displayed via auser interface of a user device for user selection of cleaning patternsfor cleaning of the open area 2904 b shown in FIG. 30B. The segmentationmaps 3102 displayed via the user interface may thus allow for selectionof a north-south rank direction 2910 a, a west-east rank direction 2910b, or a criss-cross rank direction 2910 c in the non-clutter region 2904b. A level of thoroughness of the cleaning (for example, a singlecleaning pass or multiple cleaning passes in each direction of travel)may also be selected for each of the directions of travel 2910 a-2910 c.That is, in accordance with embodiments described herein, a coveragepattern may be determined not only based on the differently-classifiedsub-regions of a surface, but also based on user-specified parameters,including direction of travel, levels of cleaning, and/or order orpriority of cleaning. User inputs labeling one or more regions of thesegmentation map may further facilitate and/or optimize cleaningoperations performed by a mobile robot as described herein.

Determining coverage patterns based on classification of clutter andnon-clutter areas and/or user-defined area cleaning operations describedherein with reference to FIGS. 29A-29D may also be based on flooringtype (for example, wet cleaning of hard floors, dry cleaning of carpetedfloors). FIG. 32 is a graphical representation of a segmentation mapview 3202 that may be displayed via a user interface of a user deviceillustrating detection and/or user selection of floor types forarea-specific cleaning operations according to some embodiments of thepresent invention. As shown in FIG. 32, the graphical representation3202 may indicate flooring types including hard floor 3204 and softfloor or carpet 3206, in combination with the detected clutter areas2906 and open areas 2904. That is, the clutter areas 2906 and open areas2904 may be further identified as hard floor open areas 3205, hard floorclutter areas 3207, soft floor open areas 3209, and soft floor clutterareas 3211, based on detection or user indication of the correspondingflooring type. Also, areas identified as clutter 2906 or open 2904 mayinclude more than one floor type 3206 or 3204, or vice versa.

For example, the lowermost open area 2904 shown in FIG. 32 includes botha soft floor/carpet area 3206 and a hard floor area 3204.

The flooring types 3204, 3206 may be detected by the robot 100 duringnavigation of the surface based on inputs from one or more sensorsthereof, and/or based on receiving identification of the floor typescorresponding to one or more regions of the segmentation map 3202 from auser device via a user interface. For example, detection of the flooringtypes 3204, 3206 may be performed by the robot 100 as described in U.S.Patent Application Publication No. 2016/0235270 to Santini, filed Feb.13, 2015, the disclosure of which is incorporated by reference herein.Also, although only hard floor 3204 and carpeted floor 3206 types areshown, it will be understood that additional flooring types, such astypes of hard floor (for example, wood or concrete) and/or types ofcarpet (for example, shag or Berber) may also be detected by the robot100 and/or specified via the user interface. Also, the type of cleaningoperations (for example, wet or dry cleaning) to be performed by therobot 100 may be determined responsive to detection or identification ofthe flooring types 3204, 3206.

User-indicated flooring types (as received, for example, via the userinterface) may also be aligned or otherwise determined to correspondwith the flooring types 3204, 3206 detected by the robot 100 based onsensor inputs and/or machine learning. For example, via the userinterface, a user may draw or otherwise define a boundary 3205 around arug 3209 in lowermost open area 3204 shown in the map 3202 to indicatethat it corresponds to a carpet area and/or a keep-out zone. The robot100 may also detect (for example, using an acoustic sensor) edges of therug 3209 on a hard floor area 3204; however, it may be determined thatthe user-defined boundary 3205 (as displayed on the map 3202) does notexactly correspond to the actual boundaries of the rug 3209 as detectedby the robot sensor(s). For example, as shown in FIG. 32, the userdefined boundary 3205 is shifted relative to the boundaries of the rug3209. In response to detecting this discrepancy, the robot 100 mayrecognize a margin of error associated with the user-defined boundary3205 based on the displayed map 3202 relative to the detected boundariesof the rug 3209 (particularly as the map 3202 may be simplified fordisplay via the user interface), and may verify its detection of thetransition between the hard floor area 3204 and the soft floor area 3206responsive to the user-defined boundary 3205. In some embodiments, theuser-defined boundary 3205 may be adjusted on the displayed map 3202 tocorrespond to the detected boundaries of the rug 3209, for example byshifting, re-sizing, or otherwise re-aligning the user defined boundary3205 on the map 3202.

Coverage patterns based on user-defined area cleaning operations asdescribed herein may be further improved by employing room-by-roomcleaning, for example, based on room segmentation operations describedherein. In some instances, room-by-room cleaning operations may allowfor cleaning time reduction of up to 50%. Also, cleaning performance maybe further improved when based on persistent mapping (for example, byupdating static versus dynamic areas of an operating environment overtime) in combination with user labeling of areas and/orcontext-sensitive behaviors. For example, in based on the determinedcoverage pattern, the robot 100 may perform edge cleaning responsive toencountering a perimeter or boundary after completing open areacleaning. As another example, after completing clutter area cleaning,the robot 100 may not traverse a previously-cleaned open area in thesame rank direction that was used to clean the open area. The operationsto compute the coverage patterns described herein may be performed bythe robot 100, the robot management server 200, and/or the user device202.

Clutter detection and classification operations described herein mayutilize the occupancy data collected by a robot 100 in combination withfeedback received from a user device in some embodiments. For example,objects (including static objects such as walls and dynamic objects suchas clutter) can be detected by the robot 100 using a classifierincluding hard-coded or programmed rules. The programmed rules may bebased on, for example, object area and eccentricity, object solidity,and/or object similarity to linear structures. However, as noted above,objects defined by pixels that appear to be attached to a detectedboundary can be interpreted as being merged into walls. Such inaccurateinterpretation of the collected occupancy data may result in generationof inaccurate maps for presentation to the user.

As such, in some embodiments, the classifier may be trained based on theoccupancy data collected by the robot 100, and further fine-tuned basedon feedback input received from a plurality of users. In particular,patterns of clutter identified by one user can be stored in a data store(for example, the data store 2106 of FIG. 21) and utilized by otherusers and/or robots, for example using convolutional neural net(CNN)-based object detection. FIGS. 34A through 34F illustrate exampleoperations for distinguishing and removing clutter from map data. Inparticular, an occupancy or grid map 3402 generated based on datacollected by navigation of the mobile robot (as shown in FIG. 34A) maybe used to detect and distinguish pixel areas 3406 representing objectsfrom pixel areas 3404 representing background areas (as shown in FIG.34B). More particularly, from the map 3402 shown in FIG. 34A, a boundary3404 shown in FIG. 34B is extracted, and the pixel areas 3406representing objects are classified as boundary-detached obstacle pixels3406 a and boundary-attached obstacle pixels 3406 b in FIG. 34C. Theboundary-detached obstacle pixels 3406 a are removed as shown in FIG.34D, such that boundary-attached obstacle pixels 3406 b remain as shownin FIG. 34E. In some embodiments, the boundary-attached obstacle pixels3406 b may be identified and classified as clutter (rather than as awall) by accessing a data store including ground truth informationgenerated at least in part from labeling inputs received from multipleusers. The boundary-attached obstacle pixels 3406 b are thereby removedsuch that only the boundaries 3404 representing walls remains in thegrid map 3402 as shown in FIG. 34F.

A data store, such as the data store 2106 described above, that may beused in identifying and classifying pixels as clutter may be generatedfrom user inputs labeling particular structures as walls (staticstructures) or clutter (dynamic structures). CNN-based object detectionmay be performed remote from the mobile robot 100, for example, by therobot management server 200 and/or other cloud server in someembodiments. The CNN classifier may be pre-trained using storedclassification data collected by many robots (for example, based onstatic maps collected from robot navigation), and fine-tuned usingstored data generated from user feedback received from one or more userdevices.

FIGS. 35A through 35D illustrate examples of patterns 3504, 3506 thatmay be stored in the data store and used for clutter classification andfeature extraction as described herein. More particularly. FIGS. 35A-35Billustrate patterns 3506 indicating clutter, while FIGS. 35C-35Dillustrate patterns 3504 indicating walls, as received and/or identifiedbased on feedback received from a plurality of user labeling inputs. Theuser labeling inputs may be received via a user interface of the userdevice, such as the user device 202 and/or the wireless electronicdevice 400 described herein. Features represented by or corresponding tothe patterns shown in FIGS. 35A through 35D may thus be learned andextracted from subsequent occupancy data collected by a robot 100 using,for example, an autoencoder. The patterns and/or shapes shown in FIGS.35A through 35D may be further classified based on additional userlabeling inputs. For example, particular ones of the patterns 3506indicating dynamic structures in FIG. 35B may be distinguished askitchen appliances rather than furniture, or vice versa, based on userlabeling of the corresponding patterns. Likewise, types of furniture(for example, chairs vs. tables) may be distinguished responsive toaccessing a data store that maintains an archive of user labeling inputsclassifying types and sub-types of clutter, which may be subsequentlyused by one or more mobile robots in recognizing such patterns in one ormore different operating environments. In further embodiments,pattern-based object detection in accordance with FIGS. 35A through 35Dmay used or combined with image-based data for image classification.

FIGS. 36A through 36C are graphical representations illustratingoperations for updating persistent map data by segmentation transfer andfeature matching between multiple maps according to some embodiments ofthe present invention. In particular, grid maps 3602 a, 3602 b of anoperating environment generated from occupancy data collected by amobile robot 100 may differ from day-to-day navigation of the operatingenvironment. These differences may arise, for example, from sensor noiseand/or differing placement or movement of clutter (including people,animals, and/or furniture) in the operating environment.

In order to maintain consistent segmentation in such a dynamic operatingenvironment, embodiments herein may transfer segmentation between mapscollected at different times and/or under different conditions.Operations for such segmentation transfer may include feature matching(as shown in FIG. 36A) and outlier removal (as shown in FIG. 36B).Feature matching may include updating a subsequently obtained map 3602 bof the operating environment with one or more features identified from aprevious map 3602 a, for example, by matching boundaries and/orcorrelating shapes that are common to the multiple maps 3602 a, 3602 bthat were generated from occupancy data detected at different timesand/or under different conditions in the same operating environment. Thefeatures 3606 that are present or persistent in multiple maps 3602 a,3602 b may be recognized as corresponding to static features, such aswalls or other non-movable structures. Outlier removal may includedeleting areas 3604 (for example, doorways and/or hallways) appearing insome of the maps 3602 a, 3602 b but not in others. Feature matchingoperations can be based both on imaging and non-imaging data collectedby the robot, for example, based on bumper hits captured by non-imagingsensors as well as images captured by the camera of the mobile robot.

Also, boundaries 3609 indicating room segmentation can be transferredbetween maps 3602 a, 3602 b that are generated based on occupancy datacollected at different times and/or operating conditions (as shown inFIG. 36C). User inputs and/or modifications to a segmentation mapdisplayed via a user interface of a user device may also be transferredin updating maps as described herein. Such maintenance of consistentsegmentation 3609 may allow users to observe the same/similar cleaningbehavior or sequence at each operation of the mobile robot, increasingpredictability of robot operation. Also, transferring segmentation 3609in updating maps as described herein may allow users to maintainpreviously-defined editing of the segmentation and/or apreviously-defined operating sequence for room-by-room or region-basedcleaning. As such, a user may observe relatively stable behavior by themobile robot despite some changes in the operating environment itself.

FIG. 37 is a graphical representation that may be displayed via a userinterface 3710 of a communication device 3700 illustrating operation ofa mobile robot responsive to user selection of temporal status accordingto some embodiments of the present invention. As shown in FIG. 37, a mapindicating navigation of the operating environment by the mobile robot100 may be generated and stored along with temporal data indicating thetimes at which the robot was at each location shown in the map. Based onthis data, a user input element 3750 (illustrated as a time slider userinterface) may be presented via the user interface 3710 of a user device3700. The user input element 3750 is configured to receive a userspecification of a time for which operating state information for therobot 100 is desired. In some embodiments, an animation 3720illustrating the location and operation of the mobile robot 100 may begenerated and displayed responsive to receiving a user selection of aparticular time or times via the user input element 3750.

More particularly, in the example of FIG. 37, the time-slider bar 3750is configured to receive a user input that manipulates slider element3751 to respective positions along a length of bar element 3755. The barelement 3755 visually represents a temporal range for which operatingstate information for the robot 100 been stored or is otherwiseaccessible. Markings indicating may be respective times may be providedalong the length of the bar element 3755. The user interface 3710 isconfigured to detect a position of the slider element 3751 along thelength of the bar element 3750 responsive to manipulation of the sliderelement or other user input, from which the device 3700 is configured todetermine the time specified by the user input based on correspondenceof the position of the slider element 3751 along the temporal rangerepresented by the bar element 3755. The user interface 3710 is therebyconfigured to display status indications representing the operatingstates of the robot 100 that are associated with temporal datacorresponding to the time specified by the user input. An animation 3720illustrating the location and operation of the mobile robot 100 over aselected portion of its run-time may also be displayed responsive touser manipulation of the time-slider bar 3750.

The animation 3720 may visually represent actual robot operating statusinformation (based on actual past or present operation of the robot 100,as previously logged or currently reported), and/or expected robotoperating status information (based on scheduled future operation of therobot 100, as specified by a user). The actual and expected/scheduledoperating states (and the past, present or future temporal dataassociated therewith) may be continually logged and updated, and may bestored in a database. For example, in some embodiments, a computingdevice (such as the robot management node 200) may receive dataindicating actual operating states (and associated past or presenttemporal data) and/or expected operating states (and associated futuretemporal data) of the robot 100. The data indicating the operatingstates and associated temporal data may be requested (or “pulled”) bythe computing device, or may be reported (or “pushed”) by the robot 100and/or network sources in communication therewith. The animation 3720 ofcleaning progress may also be presented with various granularity. Forexample, the cleaning progress on a sub-region or room level may beselected and displayed in an enlarged view, rather than as small squaresas shown in FIG. 37. Additionally or alternatively, the animation 3720may illustrate the trajectory of the robot 100, showing a previous- orreal-time path of the robot 100 responsive to user selection. In someembodiments, the time selected via the input element 3750 is associatedwith a timestamp in a video feed collected and stored by the mobilerobot 100, and the user interface 3710 may display images/video capturedat that selected time by a camera mounted to the mobile robot 100. Moregenerally, a viewable history 3720 of the instantaneous operation andoperating patterns of the mobile robot 100 throughout the operatingenvironment may be displayed at a user-specified time or over auser-specified time range.

In the above-description of various embodiments of the presentdisclosure, aspects of the present disclosure may be illustrated anddescribed herein in any of a number of patentable classes or contextsincluding any new and useful process, machine, manufacture, orcomposition of matter, or any new and useful improvement thereof.Accordingly, aspects of the present disclosure may be implemented inentirely hardware, entirely software (including firmware, residentsoftware, micro-code, etc.) or combining software and hardwareimplementation that may all generally be referred to herein as a“circuit,” “module,” “component,” or “system.” Furthermore, aspects ofthe present disclosure may take the form of a computer program productcomprising one or more computer readable media having computer readableprogram code embodied thereon.

Any combination of one or more computer readable media may be used. Thecomputer readable media may be a computer readable signal medium or acomputer readable storage medium. A computer readable storage medium maybe, for example, but not limited to, an electronic, magnetic, optical,electromagnetic, or semiconductor system, apparatus, or device, or anysuitable combination of the foregoing. More specific examples (anon-exhaustive list) of the computer readable storage medium wouldinclude the following: a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an appropriateoptical fiber with a repeater, a portable compact disc read-only memory(CD-ROM), an optical storage device, a magnetic storage device, or anysuitable combination of the foregoing. In the context of this document,a computer readable storage medium may be any tangible medium that cancontain, or store a program for use by or in connection with aninstruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device. Program codeembodied on a computer readable signal medium may be transmitted usingany appropriate medium, including but not limited to wireless, wireline,optical fiber cable, RF, etc., or any suitable combination of theforegoing.

Computer program code for carrying out operations for aspects of thepresent disclosure may be written in any combination of one or moreprogramming languages. The program code may execute entirely on a userdevice, a mobile robot, or a remote server described herein, or partlyon one or more of each. In the latter scenario, the remote computer maybe connected to the user's computer through any type of network,including a local area network (LAN) or a wide area network (WAN), orthe connection may be made to an external computer (for example, throughthe Internet using an Internet Service Provider) or in a cloud computingenvironment or offered as a service such as a Software as a Service(SaaS).

Aspects of the present disclosure are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable instruction executionapparatus, create a mechanism for implementing the functions/actsspecified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that when executed can direct a computer, otherprogrammable data processing apparatus, or other devices to function ina particular manner, such that the instructions when stored in thecomputer readable medium produce an article of manufacture includinginstructions which when executed, cause a computer to implement thefunction/act specified in the flowchart and/or block diagram block orblocks. The computer program instructions may also be loaded onto acomputer, other programmable instruction execution apparatus, or otherdevices to cause a series of operational steps to be performed on thecomputer, other programmable apparatuses or other devices to produce acomputer implemented process such that the instructions which execute onthe computer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

It is to be understood that the terminology used herein is for thepurpose of describing particular embodiments only and is not intended tobe limiting of the invention. Unless otherwise defined, all terms(including technical and scientific terms) used herein have the samemeaning as commonly understood by one of ordinary skill in the art towhich this disclosure belongs. It will be further understood that terms,such as those defined in commonly used dictionaries, should beinterpreted as having a meaning that is consistent with their meaning inthe context of this specification and the relevant art and will not beinterpreted in an idealized or overly formal sense unless expressly sodefined herein.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousaspects of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

The foregoing is illustrative of the present invention and is not to beconstrued as limiting thereof. Although a few example embodiments ofthis invention have been described, those skilled in the art willreadily appreciate that many modifications are possible in the exampleembodiments without materially departing from the novel teachings andadvantages of this invention. Accordingly, all such modifications areintended to be included within the scope of this invention. Therefore,it is to be understood that the foregoing is illustrative of the presentinvention and is not to be construed as limited to the specificembodiments disclosed, and that modifications to the disclosedembodiments, as well as other embodiments, are intended to be includedwithin the scope of the invention.

1. (canceled)
 2. A method of operating a mobile robot in an environment,the method comprising: executing, by a processor of a mobile terminal,computer readable instructions stored in a non-transitory computerreadable storage medium to perform operations comprising: displaying,via a user interface of the mobile terminal, a segmentation map of theenvironment; receiving cleaning mission information comprisingdemarcated regions on the segmentation map including one or more regionsto be cleaned and at least one keep-out region to be excluded from beingcleaned by the mobile robot; and controlling the mobile robot tonavigate the environment and to perform a cleaning operation inaccordance with the cleaning mission information.
 3. The method of claim2, wherein the computer readable instructions are executed to performoperations further comprising: receiving, via the user interface of themobile terminal, a user input identifying the one or more regions to becleaned and the at least one keep-out region to be excluded from beingcleaned.
 4. The method of claim 2, wherein the computer readableinstructions are executed to perform operations further comprising:receiving, via the user interface of the mobile terminal, a user-definedboundary of the at least one keep-out region, wherein displaying thesegmentation map includes displaying the user-defined boundary of the atleast one keep-out region on the segmentation map.
 5. The method ofclaim 4, wherein receiving the user-defined boundary of the at least onekeep-out region is responsive to a gesture on a touch-screen of the userinterface.
 6. The method of claim 4, wherein the at least one keep-outregion is defined with respect to an obstacle in the environment.
 7. Themethod of claim 6, wherein the computer readable instructions areexecuted to perform operations further comprising: displaying, via theuser interface of the mobile terminal, a boundary of an obstacle in theenvironment detected by the mobile robot using a sensor associatedtherewith; and receiving a user input to modify the user-definedboundary of the at least one keep-out region based on a discrepancybetween the user-defined boundary and the detected boundary of theobstacle.
 8. The method of claim 2, wherein the computer readableinstructions are executed to perform operations further comprising:receiving, via the user interface of the mobile terminal, respectivelabels for the one or more regions to be cleaned and the at least onekeep-out region, wherein displaying the segmentation map includesdisplaying the respective labels on the segmentation map.
 9. The methodof claim 2, wherein: the cleaning mission information includes cleaningthe one or more regions to be cleaned according to a user-specifiedorder; and the segmentation map displayed via the user interfaceincludes an indication of the user-specified order.
 10. The method ofclaim 2, wherein the computer readable instructions are executed toperform operations further comprising: determining a coverage patternfor the mobile robot indicating a sequence of navigating the demarcatedregions, wherein controlling the mobile robot to navigate theenvironment is in accordance with the sequence indicated by the coveragepattern.
 11. The method of claim 10, wherein the coverage patternincludes a rank direction indicating a direction of the mobile robottraveling in one or more regions to be cleaned while avoiding enteringthe at least one keep-out region, wherein displaying the segmentationmap via the user interface includes displaying the rank direction on thesegmentation map.
 12. The method of claim 2, wherein the cleaningmission information includes one or more cleaning schedules for cleaningrespective subsets of the demarcated regions at respective times orduring respective time intervals.
 13. The method of claim 2, wherein thesegmentation map further includes information about respective flooringtypes of the one or more regions to be cleaned.
 14. The method of claim13, wherein the segmentation map further includes information aboutrespective types of cleaning operations for cleaning the one or moreregions to be cleaned, the respective types of cleaning operations basedon the respective flooring types of the one or more regions to becleaned.
 15. A mobile robot system, comprising: a mobile robot; and amobile terminal operably in communication with the mobile robot, themobile terminal including a user interface and a controller circuit, thecontroller circuit configured to: display, via the user interface, asegmentation map of an environment for navigating the mobile robot;receive cleaning mission information comprising demarcated regions onthe segmentation map including one or more regions to be cleaned and atleast one keep-out region to be excluded from being cleaned by themobile robot; and generate a control signal to control the mobile robotto navigate the environment and to perform a cleaning operation inaccordance with the cleaning mission information.
 16. The mobile robotsystem of claim 15, wherein the controller circuit of the mobileterminal is configured to receive a user input identifying the one ormore regions to be cleaned and the at least one keep-out region to beexcluded from being cleaned.
 17. The mobile robot system of claim 15,wherein the controller circuit of the mobile terminal is configured toreceive a user-defined boundary of the at least one keep-out region, andto display the user-defined boundary of the at least one keep-out regionon the segmentation map.
 18. The mobile robot system of claim 17,wherein: the mobile robot is configured to detect a boundary of anobstacle in the environment using a sensor associated with the mobilerobot; and the controller circuit of the mobile terminal is configuredto display, via the user interface of the mobile terminal, the detectedboundary of the obstacle, and to receive a user input to modify theuser-defined boundary of the at least one keep-out region based on adiscrepancy between the user-defined boundary and the detected boundaryof the obstacle.
 19. The mobile robot system of claim 15, wherein thecontroller circuit of the mobile terminal is configured to: receive, viathe user interface of the mobile terminal, respective labels for the oneor more regions to be cleaned and the at least one keep-out region; anddisplay, via the user interface, the respective labels on thesegmentation map.
 20. The mobile robot system of claim 15, wherein: thecleaning mission information includes cleaning the one or more regionsto be cleaned according to a user-specified order; and the segmentationmap displayed via the user interface includes an indication of theuser-specified order.
 21. The mobile robot system of claim 15, whereinthe controller circuit of the mobile terminal is configured to:determine a coverage pattern for the mobile robot indicating a sequenceof navigating the demarcated regions; and generate the control signal tocontrol the mobile robot to perform the cleaning operation in accordancewith the sequence indicated by the coverage pattern.